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ABSTRACT 

This guide is intended to assist states and service 
delivery areas (SDAs) in addressing the new oversight 
responsibilities and opportunities stipulated by the Job Training 
Partnership Act (JTPA) with respect to planning a local evaluation. 
The following topics are covered in the individual chapters: the 
objectives and scope of JTPA evaluation (choosing to evaluate and 
reviewing JTPA evaluation materials and options); preliminary 
planning issues (assessing evaluability , building in utility, and 
understanding the organizational context); progression toward a JTPA 
evaluation plan (formulating evaluation questions and a research 
design, developing an evaluation plan, and estimating evaluation 
costs and benefits); and implementation issues (collecting and using 
data and funding and staffing a JTPA evaluation). A conclusion and 
suggestions for further reading are also provided. A report by Bonnie 
Snedeker entitled "A Report on a National/State Survey of Local JTPA 
Constituencies" is appended. (MN) 
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CONITEXT OF THIS VOLUME 

This is one in a series of volumes produced by the JTPA EVALUATION DESIGN PROJECT. 



PURPOSE AND PHILOSOPHY 

The purpose of this project has been to develop a set of evaluation tools that are useful f o states and u • - 
areas (SDAs) in judging the way their JTPA programs are being managed and the impact they arc %?ivv 
has been to base these analytic and managerial tools on sound program concepts and research mchou5 
them such that the mformation obtained is of practical and direct use in improving JTPA policies and pV 
state and local level. This kind of information is also expected to make a unique contribution to national ti 
and Federal oversight of JTPA. 

It is hoped that these volumes will stimulate and support state and local evaluation efforts in JTPA, and prom, more 
consistency than in previous programs with respect to the issues studied and the methods used to investigate then. An 
important goal is to encourage the generation of complementary information on program implementation and impact 
that IS comparable across states and SDAs. Comprehensive, comparable information is essential to the devei ^rn^nt of 
a valid and reliable knowledge base for resolving problems and imjDroving programs. It is also required for adjr.iing na- 
tional training strategies to changing needs and priorities at the state and local level. 

PRODUCTS 

Consistent with this purpose and philosophy, the project has produced a :.t of materials to assist states a. u ^DAs in 
evaluating their programs. These are to be useful in planning, designing and implementing evaluation activities As an 
integrated collection, each set is developed to support comprehensive evaluations over the JTPA planning cyde. 

The careful tailoring of these materials to state and local users is appropriate. JTPA represents a new employment and 
irainmg policy shaped not only by the experience of managers and the perspectives of employers, but by scientific assessments 
of previous approaches for addressing unemployment, poverty and other barriers to economic security. In this context 
the value of JTPA programs is also expected to be judged. In fact, the Act's assessment requirements are more explicit 
and sophisticated than those of any employment and training legislation to date. It clearly distinguishes between monitor- 
ing activities, whose purpose is to determine compliance (such as with performance standards) and evaluation activities 
whose purpose is to determine how a program is being managed and implemented, and the kinds of effects it is having 
on recipients and relevant others. Equally significant, new constitutencies are expected io make these more rigorous 
assessments. States and SDAs now have this important responsibility. It is the first time in the history of employment 
and training programs that the Federal government's evaluation role has been significantly reduced. 

This change affords states and local areas opportunities to influence public policy. It also requires them to assume new 
oversight responsibilities. Program evaluation is expected to become an integral part of the management of organizations 
admmistermg, planning and delivering public training services. This is as it should be. The more information available 
at these levels, where changes in organizations can most readily be made, the more effective the management of JTPA 
programs. This project was undertaken in that context. 

The evaluation tools produced by the project have been developed with a sensitivity to the differing needs, interests 
and resources of state and local users. They have been packaged into a single comprehensive and integrated set of volumes 
called JTPA Evaluation at the Stai: and Local Level. The set contains planning and evaluation guides and issue papers 
The following volumes are available in the set: 



Volume 



Author 



I: Overview 

IJ: A General Planning Guide 
III: A Guide for Process Evaluations 
III Supplement: Some Process Issues at the State Level 
IV: A Guide for Gross Impact Evaluations 
V: A Guide for Net Impact Evaluations 
VI: An Implementation Manual for Net Impact Evaluations 
VII: Issues Related to Net impact Evaluations 

A. Issues in Evaluating Costs and Benefits 

B. The Debate Over Experimental vs. Quasi-Experimental Approaches 
VIII: MIS Issues in Evaluating JTPA 

NOTE: Although each of the discrete products listed above is the responsibility of a single author, each seeks to incor- 
porate the results of professional peer review, the many excellent recommendations of the advisory group, and the ideas 
and suggestions of the numerous practitioners interviewed in the process of developing these materials 



Project Team 
Deborah Feldman 
David Grembowski 
David Grembowski 
Carl Simpson 
Terry Johnson 
Terry Johnson 

Ernst Stromsdorfer 
Ann Blalock 
David Grembowski 



To further qualify these volumes, Volume III is accompanied by a supplement for state users. This is consistent with 
the significant differences betweei: states and SDAs in the kinds of process issues that are most essential to study. The 
volume on net impact evaluations ij sufficiently technical, because of the statistical methods involved, that a practical 
manual has been written to accompany it. This guide and manual tend to be more appropriate for states, since relatively 
large sample sizes are required for analysis. However, they are equally useful to larger SDAs and consortia of smaller 
SDAs which may want to jointly study the net impact of their programs. Regional evaluations, for example, can be very 
productive in providing management information relevant to regional labor markets. Although there is a separate issue 
paper on evaluating costs and benefits, this issue is also covered in the gross impact and net impact guides. In this respect, 
the user benefits from three related but different approaches to this important element of program evaluations. Also, 
the user should be aware that the Appendix of Volume II includes A Report on a National/State Survey of Local JTPA 
Constituencies, This survey was carried out by Bonnie Snedeker, with the assistance of Brian O'Sullivan, to provide addi- 
tional input from practitioners to the development of the planning and process evaluation guides. 

In conclusion, several expectations have directed the development of these volumes: 



THE GUIDES 

The General Planning Guide 

This guide is to assist users in planning, funding and developing an organizational capacity to carry out process, gross 
outcome, and net impact evaluations and to utilize their results. Separate state and local versions are available. 

The Evaluation Guides 

These volumes are to have the following characteristics: 

□The guides are to complement one another. 

•They are to provide information on program management and other characteristics of program implementation, which 
can: 

—Describe the v/ay in which administrative, managerial and service delivery policies and practices operate to affect 

outcomes, as a set of interventions separate from the program's services. 
—Pinpoint the source, nature and extent of errors and biases for which adjustments must be made in gross and net 

impact evaluations. 
—Help explain the results of gross and net impact evaluations. 

•They are to provide information on aggregate gross outcomes, and outcomes differentiated by type of service and 
type of recipient, which can: 

—Describe relationships between certain implementation modes and service strategies, and a broad array of client and 

employer outcomes. 
—Help explain the results of net impact evaluations. 

—Suggest the more important outcomes that should be studieu in net impact evaluations. 

—Help sort out those aspects of implementation that may be most critical to study in process evaluations. 

•They are to provide information on net impact (the program's return on investment), which can: 
—Closely estimate the effect of the program's services on clients. 

—Suggest which services and client groups are most important to study in broader but less rigorous gross impact studies. 
—Help identify the decision points in program implementation (particularly service delivery) which may be most 
important to study in process evaluations. 

□The guides are to enable the user to carry out comprehensive assessments of JTPA programs. 

•They are to allow the user to acquire several different perspectives on the same program within a particular time period: 
on program implementation, on outcomes for clients and employers and on net impact. 

•They are to permit the user to interrelate these different kinds of information to gain a wider understanding of whar 
is happening in a program and why. 

□The guides are to describe approaches and methodologies as consistently as possible, to achieve comparability, 
•They are to define variables and relationships as similarly as possible, 

•They are to define research designs, and methods of data collection and analysis using as similar concepts as possible. 

□The guides are to draw from past research or. employment and training programs, as well as seek new approaches and 
methods of specific value in evaluating JTPA at the state and local level. 

•They are to replicate, to the extent possible and feasible, the issues and measures refiecred in Federal ir.cnitoring and 
evaluation decisions. 

•They are to make selective use of the results of relevant CETA studies, national studies of JTPA, and issue papers 
on JTPA evaluation by national public interest organizations in the employment and training area, 

•They are to rely on the professional literature in applied social research. 
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INTRODUCTION 



While this planning guide may be used Independently, It Is designed to 
supplement a set of evaluation guides In the series titled jtpa 
Eyjjuatlon at the State and Local level. The companion volumes to this 
guide address specific JTPA evaluation research Issues. This guide 
recognizes that technical research tools are not always all that Is 
needed In carrying out a successful JTPA evaluation effort. Local JTPA 
practitioners have a host of practical concerns about evaluation, 
ranging rrom how to promote evaluation as a worthwhile activity to how 
to hire a good consultant. The primary purpose of this guide Is to 
address these more practical concerns about planning and carrying out 
JTPA evaluation, concerns which cross-cut the various evaluation 
approaches described In the companion volumes. 

This guide begins with some thoughts about the nature, purposes and 
value of JTPA program evaluation. While students of evaluation may 
find nttie new here, the Ideas presented may be helpful to the 
non-specialist or to administrative decision-makers who need to know 
more about evaluation before they can support It within their 
organization. The Introductory portion of the guide Is also designed 
to familiarize the reader with the various evaluation materials 
available through the several volumes comprising JTPA Evaluation at the 
State and Local Level. From this preliminary section the reader should 
come away with a sense of how all these materials fit together and how 
they may be used to conduct various kinds of jtpa evaluations at the 
SDA level. 

The middle portion of this guide (Sections Two and Three) develops an 
overall planning context for carrying out JTPA evaluation. As much as 
possible, planning Issues are presented within a roughly sequential 
framework. The framework begins with an examination of what Is 
organizationally possible (what are the organizational supports for and 
constraints to evaluation) and ends with an assessment of evaluation 
costs and benefits. 

Some areas of evaluation planning are less amenable to assignment and 
discussion within a sequential framework. The final chapters of this 
guide (Section Four) are devoted to important resource planning topics 
which deserve separate treatment. Those topics Include funding 
concerns, staffing needs and options, and data collection issues. 

In producing this guide, the assumption is that the potential audience 
of JTPA administrators, practitioners and evaluators is wide-ranging in 
terms of technical background and information needs. Such a guide 
always runs the risk of being too simplified for some and too cursory 
for others. As much as possible, this guide adheres to a middle 
course: It examines the basic evaluation planning and implementation 
issues within the specifi c context of JTPA . but in honoring the 
diversity of its readers' interests and needs it does not offer a 
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econd guide is on state JTPA evaluation Issues and concerns Some 
local practitioners may also be Interested In this additional 
perspective on JTPA evaluation. aaaitionaJ 

1nterv^^ew^^^n^''''^^nfT^^"J?'^^ ^^^^^ conected through 

ntervlews and Informal discussions with numerous federal state and 
local JTPA practitioners, administrators, and evaluators A most ai? 
of the specific examples of evaluation experiences and activ ties arl 
derived from these Important Informants. activities are 
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SECTION 1 

AN INTRODUCTION TO JTPA EVALUATION 



These first chapters are addressed to a broad audience of JTPA administrators 
and practitioners who must decide whether or not to evaluate and how to evaluate 
JTPA. The first chapter tackles the question "Why evaluate?", setting forth some 
specific rationales for evaluating JTPA programs at the state level. In address- 
ing the concern "How should we evaluate?" the second chapter describes the 
various JTPA evaluation materials and approaches contained in the set of 
volumes this guide accompanies. 
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CHAPTER 1 
CHOOSING TO EVALUATE 



What Is Evaluation? 

Evaluation vs. Monitoring 
Evaluation Approaches 

How can JTPA benefit from evaluation? 
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CHAPTER 1. CHOOSING TO EVALUATE 



INTRODUCTION 

JTPA decision-makers at the service delivery area (SDA) level face 
tough choices In allocating scarce program resources. JTPA 
administrative monies are restricted and no federal funds have been 
specially earmarked for SDA evaluation Initiatives. As a result, 
evaluation activities must compete for recognition against other worthy 
program Investment choices. If evaluation Is to be Incorporated Into 
the SDA program agenda, JTPA administrators and policy-makers must be 
convinced that evaluation, as a program Investment, yields significant 
management returns. This chapter Introduces the concept of evaluation 
and argues the merits of Incorporating evaluat^n activities Into JTPA 
programs . 

WHAT IS EVALUATION? 

This volume Is about planning and carrying out JTPA evaluation 
activities. Since "evaluation" has come to mean different things to 
different users and has often been loosely applied to an^ program 
assessment activity, we first must define the term. As it is used in 
this guide, evaluation refers to the systematic collection , analysis 
and reporting of information on a particular set of program activities 
and outcomes that de cision-makers wish to know more about . 

Encompassing a variety of research methods, evaluation seeks to 
determine the efficiency and effectiveness of a given program. 
Effectiveness concerns the extent to which a program , through various 
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treatments or service Interventions has met Its Intended goals . As 
outlined 1p legislation, the three principle goals of JTPA are to (1) 
Increase stable employment, (2) Increase earnings, and (3) reduce 
welfare dependency of economically disadvantaged and dislocated 
workers. In the JTPA context, then, a central question evaluation 
poses Is "how effectively are programs contributing to changes In 
employment, earnings and welfare status of the Intended target group?" 

By efficiency, we mean how well a program has used available resources 
to achieve Its Intended goals. In determining the efficiency of JTPA 
program efforts, evaluation activities might focus on the various costs 
and benefits of the program and how s;uch measures compare with those of 
other JTPA program strategies. Since In most state settings, JTPA 
resources are terribly limited, determining what Is an efficient use of 
those resources Is a particularly relevant undertaking. 

While concepts of efficiency and effectiveness are Interrelated, the 
one does not necessarily follow the other. A program may be 
tremendously efficient, yet not terribly effective, and vice-versa. 
For example, a JTPA program may be quite cost-efficient In placing a 
large number of participants, but the program's true Impact 
(effectiveness) may actually be negligible; the participants may have 
done just as well on their own without the program. 

Evaluation and JTPA 

In measuring efficiency and effectiveness, evaluation can consider both 
JTPA program processes and outcomes .. As Illustrated below, outcome 
evaluations focus on the end benefits derived from program activities; 
process evaluations focus on the activities themselves: 



Not everyone subscribes to a goal -oriented basis for evaluation. 
See Scrlven, "Pros and Cons of Goal Free Evaluation." Evaluation 
Comment , Vol. 3, No. 4 (Dec, 1972), pp. 1-4. 




OUTCCME VS. PROCESS EVALUATIONS 



Evaluation Tvoe Q uestions Asked 

Outcome Old JTPA participants benefit 

from the program? 

What kind of benefits were 
derived? 

Which participants benefitted 
most? 



How was the program Implemented? 

Which program elements contrib- 
uted to or detracted from 
achievement of program goals? 



Together process and outcome evaluations can provide a wide range of 
Information to pic members, SDA administrators, and program staff, 
allowing them to make more Informed judgments about their programs. 
More specifically, comprehensive evaluation can Inform these 
decision-makers In two ways: decision-makers (1) can better discern to 
what extent major legislative goals for JTPA are or are not achieved 
and (2) can more fully understand how JTPA programs operate In order to 
better meet program goals and Improve compliance with performance 
standards. 



Evaluation vs. Monitoring 

Sometimes evaluation Is treated as If It were an elaborate extension of 
program monitoring activities. However, the evaluation process, while 
often utilizing data collected by a monitoring system, can be viewed as 
conceptually distinct from monitoring. Within the overall jTPA 
planning, management and policy framework, evaluation and monitoring 
activities should ask different questions and serve different 
purposes. As discussed above, evaluation poses questions about how 
efficiently and effectively JTPA program goals are being met or how 
they might be better met In the future. A useful evaluation permits 
decision-makers to make judgments about the value of JTPA programs (or 
particular JTPA program aspects). 

16 
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wnat Kind of benefits were 
derived? 

Which participants benefitted 
most? 



Process How was the program implemented? 

Which program elements contrib- 
uted to or detracted from 
achievement of program goals? 



Together process and outcome evaluations can provide a wide range of 
information to Pic members, SDA administrators, and program staff, 
allowing them to make more informed judgments about their programs. 
More specifically, comprehensive evaluation can inform these 
decision-makers in two ways: decision-makers (1) can better discern to 
what extent major legislative goals for JTPA are or are not achieved 
and (2) can more fully understand how JTPA programs operate in order to 
better meet program goals and improve compliance with performance 
standards. 

Evaluation vs. Monitoring 

Sometimes evaluation is treated as if it were an elaborate extension of 
program monitoring activities. However, the evaluation process, while 
often utilizing data collected by a monitoring system, can be viewed as 
conceptually distinct from monitoring. Within the overall jtpa 
planning, management and policy framework, evaluation and monitoring 
activities should ask different questions and serve different 
purposes. As discussed above, evaluation poses questions about how 
efficiently and effectively JTPA program goals are being met or how 
they might be better met in the future. A useful evaluation permits 
decision-makers to make judgments about the value of JTPA programs (or 
particular JTPA program aspects). 
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JTPA?" we conclude with five more specific arguments for evaluating 
JTPA. 



1. Evaluation as a Mechanism for Accountability 

With shrinking public resources have come increasing demands for 
viable methods to ensure that program funds are being wisely 
spent. A program's worth must be demonstrated not only to 
elected officials and the taxpaying public, but also to users of 
program services and their advocates. Evaluation offers a 
rational management tool for examining the value of JTPA 
programs and ensuring accountability to these interested parties. 

The language of fisca l accountability is built into JTPA 
legislation (Job training is an "investment" in social capital), 
but evaluation is not limited to answering questions about the 
fiscal costs and benefits of JTPA. Evaluation also speaks to 
questions of equity. If JTPA is not targeting services to the 
people the legislation was designed to assist, equity is not 
being served. Evaluation, then, can be used as a tool to 
provide socia l as well as fiscal accountability to the public. 

2. Evaluation as a Planning and Management Tool 
Increasingly, program operators are viewing evaluation as a 
management tool for improving decision-making. Under JTPA, SDA 
managers may face a full spectrum of administrative tasks, 
ranging from the SDA that primarily coordinates and oversees 
program activities to the SDA that designs, staffs, and carries 
out such activities. Whatever roles the SDA (and its PIC) may 
play, evaluation's utility can be direct and immediate. 
Evaluation may help to identify gaps and overlaps in program 
services, uncover special problems in service delivery, and 
answer basic questions about the effectiveness of various 
programmatic strategies. Moreover, evaluation results may guide 
decisions about the distribution of scarce resources among 
competing program demands. 

3. Evaluation as a Policy Tool 

Evaluation can offer more long-range benefits by informing PIC 
members and other policy-makers about areas of program 
implementation and program accomplishments that are of 
importance to them. When used as such a policy tool, evaluation 
can lay the foundation for better, more innovative policy 
approaches to service delivery. 

What are JTPA policy concerns? While each local setting is 
unique, all policy-makers must generally be concerned with 
whether JTPA programs are responsive to the Act's purposes and 
local needs. Are more specific policies needed to enhance 
service provider coordine-tionV Encourage certain kinds of 
innovative efforts? Better serve target populations? Encourage 
more quality placements? Evaluation is a mechanism for 
answering these and other policy questions so that 
decision-makers can make more informed judgments about JTPA. 
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Evaluation as an Educative Process 

In a general sense, evaluation 1s Intrinsically an educative 
process because the goal of every evaluation Involves Increased 
understanding of a program on the part of someone-- program 
operators, planners, administrators, policy-makers, the public. 
However, evaluation may also be an educative process In a more 
specialized sense b^ providing an, organized way for people on 
the front lines o£ service delivery to share what they do with 
other concerned parties and vice-versa . 

This kind of Information sharing Is particularly Important In a 
complex, multi-layered service delivery system as Is found In 
the more decentralized JTPA programs (1 .e. , where service 
delivery Is contracted out). For example, the rich and complex 
results of a process evaluation may allow PIC members, elected 
officials, state administrators and the public to more directly 
grasp the complexities of effectively managing an employment and 
training program. Those outside service delivery can gain a 
better appreciation for the difficulties In delivering JTPA 
services, given the resource constraints, coordination demands 
and organizational obstacles many service providers face. 
Similarly, evaluation offers state administrative staff and SDAs 
an opportunity to communicate more effectively with each other 
about their separate concerns. 

Evaluation as a Tool for Moving Beyond Performance 
Standards 

From Its Inception, JTPA has focused attention on one type of 
performance assessment: performance standards . Since the 
standards are mandated and are to be unl formly appl led across 
states and their SDAs (unless states choose to develop their own 
regression model for performance standards), why be concerned 
about other assessment measures which are not so explicitly 
called for In legislation? Evaluation tools are a necessary 
complement to performance measures for several reasons: 

• Evaluation Helps Explain Performance . 

Consistently low performance outcomes or Inconsistent 
outcomes may clue us In that there 1s a problem, but tell us 
1 1 ttle about what Is Influencing such performance. 
Performance standards do not tell us what Is or Is not an 
effective program element. To answer these kinds of 
questions, we may use a process evaluation to systematically 
examine specific program factors. 

• Evaluation Looks at Distributive Outcomes . 

High or low performance outcomes may mask other less obvious 
distributive outcomes for clients. Who Is really being 
served by JTPA? Is It the Intended target group? Are 
clients receiving truly beneficial services and placements? 
Evaluation can directly address these distributive Issues; 
performance measurements can do so only In a limited fashion. 

• Evaluation Measures Program Impacts . 

Performance measures alone cannot answer the Important 
question "Did the program have an Impact In giving people 
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durable jobs increasing their earnings and reducing their 
vulnerability to poverty?" while a given program may boast a 
70% placement rate, we have no idea if the program's efforts 
were truly responsible for those placements or whether, in fact 
participants would have gotten the same Jobs even if they had 
not participated in JTPA. Performance outcomes need to be 
supplemented v.^ith other evaluation techniques that help sort out 
extraneous influences from the true effects of the program 
itself. In some instances, evaluation may reveal that a 
program with low performance measures is still very effective 
because it significantly impacts a target group of dif f icult-to- 
serve clients who, without the program, would otherwise not have 
been successfully trained and placed. 

CONCLUSION 

Congress intended JTPA to be a "performance- driven system" in which 
the program's measured accomplishments in training and placing 
participants would be the hallmark of program success. 

In such a decentralized federal setting, it makes sense to develop 
performance standards and reporting requirements .to ensure 
a measure of program accountability to federal authorities. But 
successful compliance with one's assigned numerical goals is only 
one source for judging the value of a program. Every SDA has its 
own set of problems, concerns and information needs particular to 
its local setting which are not necessarily addressed through 
performance standards ratings. Evaluation offers other important 
sources of information which help JTPA decision-makers to see the 
complexity of the program and to make more accurate assessments of 
its true impact on participants. Armed with such information, thost- 
decision-makers are then in a better position to develop strategies 
for further program improvement. 



What evaluation course make sense for an SDA to pursue in order to 
capture the benefits of evaluation described in the preceding 
pages? The following chapter delves further into the specific kinds 
of evaluation approaches and options available through additional 
guides in the set Evaluating JTPA at the State and Local Level. 
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CHAPTER 2 

REVIEWING JTPA EVALUATION MATERIALS AND 
OPTIONS 

What Evaluation Materials Are Available? 
What Is a Gross Impact Evaluation? 
What Is a Net impact Evaluation? 
What Is a Process Evaluation? 
How do These Evaluation Approaches Complement Each Other? 



CHAPTER 2. 



REVIEWING JTPA EVALUATION MATERIALS AND OPTIONS 
INTRODUCTION 

With the passage of JTPA in 1982, Congress created a new Isgislative 
context for planning and Implementing this country's employment and 
training programs. The new context Includes a much enhanced 
administrative and planning role for state government while allowing 
for a fair amount of policy and program initiative at the local level. 

Federal administrataive responsibilities, including evaluation, have 
dramatically receded. Previously, the federal government formulated 
evaluation policy, funded new evaluation research efforts and 
disseminated findings. Now states and SDAs must take on new oversight 
responsibilities, having relatively little experience in evaluation 
policy-making, design and implementation. The materials described here 
are part of a research effort to assist SDAs and states in carrying out 
these new roles and responsibilities. 

THE JTPA EVALUATION DESIGN PROJECT 

This planning guide is one in a series of related evaluation materials 
produced by the JTPA Evaluation Design Project.^ In this 
chapter we will briefly describe the Prolect's purposes and 
orientation, present the various materials available through the 
project and outline how JTPA administrators, planners and policy-makers 
can effectively use these materials. 

A primary purpose of the project is to create evaluation materials 



For a synopsis of the Project and its funders and participants, see 
the Preface. 
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which are useful to states and SDAs 1n planning and carrying out JTPA 
evaluation activities. A secondary purpose Is to develop several model 
evaluation strategies which, when applied across states and SDAs, can 
produce comparable Information. If SDAs, for example, use a consistent 
research strategy for assessing JTPA program Implementation, the 
lessons learned from such a process evaluation are more likely to have 
broader significance. Informing policy-makers at the federal and stat?, 
as well as local level . 



WHAT EVALUATION MATERIALS ARE AVAILABLE? 

In order to meet different users' needs, the evaluation materials 
developed by the project consist of a set of complementary volumes on 
JTPA evaluation entitled JTPA Evaluation at the State and Local Level. 
These volumes can be used Independently or In conjunction with each 
other. The set of materials described In this chapter Includes the 
following volumes: 

Volume I: Overview 

Volume II: A General Planning Guide 

Volume III: A Guide for Process Evaluations 

Volume IV: A Guide for Gross Impact Evaluations 

Volume V: A Guide for Net Impact Evaluations 3 

Volume VI: An Imple.tK ntatlon Guide for Net Impact Evaluations 

Volume VII: Issues Related to Net Impact Evaluation 

a. Issues In Evaluating Costs and Benefits 

b. The Debate Over Experimental Vs. Quasi- 
Experimental Design 

Volume VIII: HIS Issues in Evaluating JTPA 

This set of volumes is designed to offer state and local level users a 
fairly selective, yet comprehensive menu of technical assistance 
products to meet a variety of evaluation needs. Taken together, these 
products support comprehensive evaluations over the JTPA biennial 
planning cycle. However, users may also wish to selectively choose 
from this menu in order to meet particular evaluation Interests, needs 
and resources. To give a sense of the uti llty and scope of these 
materials, the various volumes are briefly described as follows. 



Because of Its cost and research requirements, the net Impact 
approach Is most likely to be used at the state level. However, 
some large SDAs or consortium of SDAs, perhaps In conjunction with 
state evaluation activities, may also wish to explore net Impacts 
of JTPA. For this reason, the description of net Impact has been 
Included here. 
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Overview (Volume I) 



This sufftnary volume outlines the materials comprising the set of 
volunies JTPA Eval uation at the State and Local Level . in condensed 
form. It covers the specific evaluation questions, research Issues and 
methodological concerns addressed In each of the companion guides In 
the total series. 

A General Planning Guide (Volume ll) 

This vo'iume provides an overview of the various evaluation tools 
available In Volumes III through VII and how these tools may be used In 
a complementary fashion. Additionally, the volume focuses on practical 
planning and Implementation Issues that cross-cut various evaluation 
designs, such as how to develop the organizational capability for 
evaluation and how evaluation activities at the state level might be 
planned, funded and carried out. Both a state and a local version of 
this guide are available. 

A Guide for Process Evaluations (Volume ill) 

A Guide for Gross Impact Evaluations (Volume IV) 

A Guide for Net Impact Evaluations (Volume V) 

iJiesA analysis guides present three distinct approaches and related 
riiethodologles for analyzing and carrying out JTPA program evaluation. 
(A specific discussion of the uses and complementary Interaction of 
these designs follow later In this chapter.) Each guide contains these 
components. 

1. A framework for analyzing either JTPA program activities 
(process evaluation) or outcomes (net and gross Impact 
evaluations). Including the specific types of evaluation Issues 
each approach addresses and the kinds of variables, measurements 
and data sources each approach requires. 

2. A discussion of research methodology. Including: 

• A recommended research design approach for answering a key 
set of evaluation questions. 

• A description of data collection and analysis methods 
covering potential pitfalls, problems and possibilities. 
Including recommendations for the use of HIS data elements 
and other data bases, where relevant. 

3. An appendix to the guide providing additional references 
and/or technical Information relating to each approach. 



Each of these analysis guides may be used Independently, In conjunction 
with each other, or with Volume VII and VIII. To the degree possible, 
the guides present Information In a straight-forward, non-technical 
fashion^ In an effort to make the presentation accessible to a wide 



Because of the research design requirements of the net Impact 
approach, the net Impact evaluation guide, of necessity, contains 
more technical Information than the other two guides. 
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audience of potential users. The development of these products has 
also been Influenced by project concerns that the materials be attuned 
to evaluation Issues of greatest Interest to users, be realistically 
Implementable In terms of research cost and complexity, and be 
committed to scientifically sound research strategies. 

An Implementation hanual for Net Impact Evaluations 
(Volume VI) 

The volume on net Impact evaluations Is sufficiently technical, because 
of the statistical methods Involved, that this practical manual 
accompanies It. 

Issues Related to Net Impact Evaluation (Volume VII) 

• Issues in Evaluating Costs and Benefits 

• The Debate over Experimental Vs. Quasi-Experimental 
Design 

This first Issue paper describes the rationale and procedures for 
estimating JTPA program costs, showing how costs and benefits are 
related In a hurrnn capital Investment framework. The second paper 
examines the pros and cons of two different net Impact research 
strategies. 

MIS Issues in Evaluating jTPA (Volume VIII) 

This Issue paper Is designed to assist users In better understanding 
how JTPA and other employment and training management Information 
systems can efficiently support evaluation. 

Which of these analysis tools will best serve the evaluation needs and 
capabilities of an Individual SDA? Given various resource constraints, 
which kind of evaluation approach should take priority? The answer 
depends In large measure on the kinds of policy priorities your state 
and PIC have ^established and the evaluation questions of greatest 
relevance to local JTPA planners, administrators, and policy-makers. 



The remainder of the chapter outlines the principle features of the 
three major evaluation approaches, the strengths and limitations of 
each approach and the key questions about JTPA programs each 
addresses. We begin with a look at evaluations which focus on outcomes. 



WHAT IS A GROSS IMPACT EVALUATION? 

In general, evaluations of the outcomes of a program are designed to 
analyze various short-term and long-term accomplishments In the context 
of the program^s stated qoals. Outcome evaluations , as the name 
Implies, focus on the end products of the program~-1n this case, 
measures of those e^^ployed, the^r wages, their status with respect to 
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the welfare system, and program costs. (Other outcomes can also be 
measured: additional client outcomes, employer outcomes, or more 
general societal outcomes, such as taxpayer dollars saved). Program 
outcome measures, taken by themselves, (without comparing them to 
outcomes for similar Individuals who do not receive JTPA-lIke services) 
can be considered gross outcomes. 

Gross Impact evaluation provides a systematic way to describe 
post-program outcomes and to analyze how service delivery alternatives 
Influence them. The gross Impact approach can be used at the state 
level to study outcomes across SDAs or at the local level to study 
outcomes within a single SDA. ^ The distinctive feature of a gross 
Impact evaluation Is Its exclusive focus on outcomes related to program 
participants: there Is no comparison or control group of 
non-participants to provide a yardstick against which overall program 
outcomes may be assessed. 

Because no untreated control group Is utilized, the gross Impact 
evaluation cannot explain participant outcomes in terms of the 
P''°fl''^'"'^ efflcacjt. In using this approach to evaluate JTPA programs, 
we do not know to what extent the outcomes are the product of other 
external Influences, such as changes In the economy, varying client 
characteristics, client use of non-JTPA training and educational 
programs or chance. In other words, we cannot differentiate between 
Impacts caused by the program and results that would have occurred In 
its absence. 

While unable to address the singular Impact of JTPA programs, the gross 
Impact approach offers some distinct advantages: 

• The research design may be less complex and easier to Implement 
than a net Impact design. 

• The approach offers a fairly quick turnaround time for information 
resu Its . 
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Prov d ng the SDA's client base is large enough to create a 
sufficient study sample. Volume iv d'iscusses sample size and other 
research considerations for state and local users. 

IS 
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A wide range of key variables may be measured, allowing for a 
richer understanding of the program's performance. 



The relative Impacts of different service strategies may be 
assessed. 



Host Importantly, In addition to the above-mentioned characteristics, 
the gross impact evaluation provides a framework for answering key 
questions about service delivery strategies, program types and employer 
and trainee post-program experiences with JTPA. In turn, these answers 
may Inform policy-making and program planning at both the state and 
local level. Some of the central questions a gross Impact evaluation 
can address are framed below: 



GROSS IMPACT EVALUATION QUESTIONS 



General Follow-Up : What Is the overall picture of participant 
employment, wages, and welfare status at some distinct time 
period after termination? How does the picture change at 
three months, six months, nine months? 

Employer Outcomei> : How do JTPA trainees Impact employers? To 
what degree does employer participation 1n JTPA raise or lower 
company turnover rate, affect training time, supervision or 
hiring? 

Comparl son of Treatments : * which treatment strategies 
(e.g., long-term vs. short-term, OJT vs. classroom training) 
have more positive outcomes relative to other treatment 
strategies? 

Comparison of Treatments Across Different Client Groups : * 
Xhlch treatment strategies are most effective for different 
client sub^populatlons, relative to other treatment strategies? 

Quality of Placements : Do post-program jobs for JTPA clients 
resemble primary, as opposed to lower quality, secondary labor 
market positions? Are positions training-related? 

if 

Note: The methodology used In the guide to answer this 
question Is referred to as "differential gross 
Impact analysis." 



While a gross Impact evaluation can answer questions about the 
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relative merits of different JTPA program components, to find out about 
the true effectiveness of JTPA we have to turn to a different kind of 
outcome evaluation, the net impact evaluation . 

WHAT IS A NET IMPACT EVALUATION? 

In contrast to gross impact, a net impact evaluation attempts to sort 
out specific program impacts from other influencing factors. A net 
impact evaluation more precisely answers the question "was the program 
effective?" by analyzing the extent to which outcomes were due 
specificallY to program treatment s rather than to other factors, such 
as participant characteristics or the environment in which the program 
operates . 

Of necessity, a net impact evaluation approach requires a complex 
theoretical base and fr^y require a larger sample size than the gross 
impact evaluation. The hallmark of the net impact research design is 
the inclusion of a comparison group of non-participants whose 
performance establishes a baseline against which JTPA client outcomes 
may be Judged. The question, then, really becomes "Do JTPA clients do 
significantly better in the labor market than non-participants with 
similar economic and educational profMes?" 

Some potential limitations of the net Impact evaluation are its greater 
design complexity and special data requirements: data elements 
required from non-JTPA sources may be difficult to access or 
unavailable. In most cases, the evaluation will be limited to the 
study of a small set of key variables and outcomes.^ However, as a 
balance to these limitations, the net impact design offers a powerful 
evaluation tool-a tool that allows us to identify more direct causal 
links between JTPA service and client outcomes, thus permitting 
stronger policy conclusions. 
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The net impact approach developed in this series uses a 

dP^?nn^"'rnr"V^ I' °PP°"^ *° ^ ^ue experimental 

design. Comparison group members are statistically matched to the 
experimental group of self-selected JTPA participants rather than 
an participants being randomly assigned to either group. Thus 

^Kmit"!L* ?fi JPP'"°«^^ "»ke additional validity assumptions 

about what is being measured. These additional assumptions may be 
viewed as a limitation imbedded within the design. 
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In assessing the effectiveness of 
not only how effective JTPA Is 
different program strategies are 
additional questions whlcii a net 
include: 



JTPA, a state or SDA may wish to know 
In general, but also how effective 
for various client subgroups. Some 

Impact evaluation of JTPA can addiess 



NET IMPACT EVALUATION QUESTIONS 



• Which program types have a greater Impact on 
client earnings? 

• Is long-term training more effective than 
short-term? 

• Are multi-strategy program approaches more 
likely to have a greater Impact thai) single 
strategy programs? 

• Do some client groups benefit more from 
certain types of training than other client 
groups? 



WHAT IS A PROCESS EVALUATION? 

By definition, outcome evaluations tell us primarily about program 
results. Examination of the factors which contribute to or help 
exfilaln those results Is more the province of process evaluations .^ 

Is a JTPA program underperf orming because of the services provided to 
clients or because of the way services are delivered? In order to 
provide Insights into wh^ a program Is achieving particular results, a 
process evaluation Illuminates the organizational manner In which the 
program Is carried out. How are services assigned to target 
populations? How are client flows organized? How are program 
functions carried out and Inter-program coordination accomplished? In 
responding to these sorts of questions, a process evaluation can reveal 
Important influences that program Implementation factors have on 
program outcomes. 



^ eJlluaViTn*^'"^' referred to as Implementation studies or formative 
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For example, how an SDA organizes Us outreach and Intake procedures 
may Intentionally or unintentionally affect which kinds of clients 
enter the JTPA system and what kinds of services they recelve-the 
selection procedures ultimately shaping employment, earnings and 
welfare savings. At the state level, a JTPA process evaluation will 
attempt to sift out those administrative and coordination arrangements 
which appear to have the most Influence on the nature and quality of 
service provision. Identifying which arrangements are contributing to 
goal achievement and which are inhibiting It. 

With the possible exception of single SDA states, JTPA Implementation 
encompasses two separate but Interrelated organizational levels, the 
state administrative level and the local service delivery level. 
Therefore, process evaluation at the two levels will be distinct from 
each other (although state decision-makers may be concerned with 
assessing Implementation at both levels), posing different questions 
about Implementation of JTPA. Some of the key questions posed by the 
state level and SDA level process evaluations are framed below: 



PROCESS EVALUATION QUESTIONS 



State Level Process E valuation SDA Level Process Evaluatinn 
How are state JTPA policies 
being formulated and Implem- 
ented? 



How are state policies and 
procedures affecting JTPA 
service delivery? 

How might communication and 
coordination between state 
agencies, states and SDAs, 
and states and federal JTPA 
administrators be Improved? 



What are the service goals 
of the SDA? Do these goals 
mesh with state employment 
and training goals? With 
JTPA goals? 

How are service delivery 
arrangements affecting who 
receives services? 

Are certain service delivery 
arrangements supporting or 
Inhibiting achievement of 
JTPA goals or particular 
state and SDA goals? 



In answering these kinds of questions about JTPA organizational 
arrangements, the process evaluation must rely on a number of data 
sources. Including less quantifiable data gathered from observation 
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OVERVIEW OF PROCESS, GROSS IMPACT AND NET IMPACT EVALUATIONS 



PROCESS EVALUATION 



A tool for studying the way JTPA \i 
being Implencnted, and the Influence 
firpletnentatlon processes art having 
on client outcomes. 



QUESTIONS ASKED: Hcv are the major 
Implantation characteristics of 
the program (which are expected to 
produce positive outccmci) influ- 
encing outcomes? Are they working 
as planned? 

BENEFITS OF THE INFORMATION: Often 
U Is the progr«Bn-s features that 
are affecting outcomes more than 
th^ services provided, process 
Infonnatlon helps the user pinpoint 
the differential effects of service 
treatments vs. the way the program 
Is being carried out. 

LINITATIONS OF THE INFORTIATION: 
Prvcess Information Is often diff- 
icult to quantify, and therefore 
the inferences are more subjective. 
Nevertheless significant clues to 
relationships between processes and 
and outcomes are possible. 



DISTINCTIVE FEATURE: The user can 
Identify those elements of Implem- 
entation that are contributlnp to 
goal achievement, or Inhabiting It, 



GROSS HVACT EVALUATJON 



A tool for studying gross outcomes 
for clients and employers: For 
all clients; for different client 
groups; for clients receiving 
different service Interventions. 



QUESTIONS ASKEO: unat are post 
program outcomes for clients (and 
employers) who experience JTPA? 
What service strategies produce 
the most positive outcomes relative 
to all other strategies? 

BENEFITS OF THE INFORMATION: States 
and SDAs can track the kinds of out- 
comes that characterize different 
groups given different services, 
without collecting Information on a 
comparison group. 



LIMITATIONS OF THE INFORMATION: 
In Interpreting the Information, we 
can not attribute any of the out- 
comes to the program Itself. Ue 
can only say that the outcones are 
occurring, due to a potential range 
of influences, one of which Is the 
program. 

DISTINCTIVE FEATURE: The user can 
obtain Information on a rich range 
of outcomes for both clients and 
JTPA employers, not available 
through the net Impact model. 



NET IMPACT EVALUATION 



A tool for studying the net impact of 
the program on clients; For all clients 
for clirnts receiving different service 
Interventions; for different client 
groups— utilizing a comparison group 
to control for non-program Influences 
on outcomes. 

QUESTIONS ASKED: Of the key outcctcs 
In the legislation, which outcomes 
can be attributed to JTPA, rather than 
to other Influences or to chance? What 
service strategies are most effective 
for which subgroups of clients? 

BENEFITS OF THE INFORMATION: The user 
can sort out which outcomes are due to 
the service Interventions, rather thM 
to other causes, consequently, the 
user has a meacjre of return on the 
Investment. 



LIMITATIONS OF THE INFORMATION: 
Because of data availability, only a 
small set of hey outcomes can be 
studied. 



DISTINCTIVE FEATURE: Policy makers can 
more truly judge the effectiveness of 
JTPA programs and service strategies. 
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The wider array of outcome measures In the gross Impact approach 
may be merged with net Impact data and used to help explain net 
Impact findings. For example, gross outcome measures may 
Include Information on quality of placement and the job 
satisfaction of the JTPA client. Linking such measures to net 
Impact findings may help answer questions about the relationship 
between quality of placement and long-term earnings or job 
retention. 



CONCLUSION 

The purpose of this chapter has been to Introduce the various 
evaluation tools contained In the set of guides entitled JTPA 
Evaluation at the St ate and Local Level , while each guide may be used 
Independently, the guides are designed to complement one another; taken 
as a totality they offer a comprehensive view of JTPA evaluation Issues 
and approaches. In particular, the three major evaluation approaches 
designated as net Impact , gross Impact and process can Interact and 
Inform each other in significant ways. Setting aside for a moment 
concerns about choosing an evaluation approach (or approaches), we now 
examine some preliminary evaluation issues which cross-cut the various 
approaches . 
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SECTION 2 
PRELIMINARY PLANNING ISSUES 



The following chapters cover some preliminary planning issues to be considered 
early on in a JTPA evaluation planning effort. While these issues are presented 
within an overall temporal framework, they do not translate easily into a set of 
discrete sequential planning steps to be set down in a guide. Rather, the plan- 
ning issues, which for clarity s sake are discusse!l here under separate category 
headings, in actual practice blend and overlap extensively with one another. 
These early evaluation considerations have long-range implications for the plan- 
ning and implementation work that occurs at later stages. While these chapters 
do not offer a defined set of planning steps, they contain numerous strategy 
considerations, check-lists, and suggestions for beginning the JTPA evaluation 
process. 
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CHAPTER 3. 



ASSESSING EVALUABILITY AND BUILDING IN UTILITY 
INTRODUCTION 

Even before specific evaluation questions are delineated or an 
evaluation approach settled upon, some Important preliminary planning 
Issues must be considered. This preliminary planning work revolves 
around three Interrelated questions concerning the setting In which the 
evaluation occurs: 

• What kind of evaluation Is feasible? 

• To what extent will the evaluation be utilized? 

• How does the organizational context Impact evaluation? 

How these questions are dealt with will have long-range consequences 
for the Implementation of the evaluation and Its ultimate Integrity as 
a useful planning, policy and management tool within JTPA, This 
chapter examines the first two questions; the following chapter 
continues with the third question. 

WHAT KIND OF EVALUATION IS FEASIBLE? 

Before fully embarking on an evaluation plan, evaluators should 
consider the feasibility of evaluating a particular JTPA program. Are 
some kinds of evaluation efforts more likely to succeed than others? 
Is the timing appropriate, or would an evaluation yield better results 
at a later date? 

To answer these kinds of questions, Rutman and others suggest that 
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evaluation planners begin with an "evaluablllty assessment" of the 
program In question. Such a preliminary assessment will help an 
organization to: 



Define the appropriate scope and timing for an evaluation 

Avoid wasting time and planning efforts that will not produce 
useful results 

Identify barriers to evaluation that need to be removed before 
evaluation can take place 

Lay the groundwork for doing further evaluation planning when 
circumstances are more conducive to such efforts 



Rutman has outlined In detail step-by-step procedures for assessing a 
program's evaluablllty. Some of the major points he and others have 
made are summarized here In terms of (1) technical factors; (2) program 
features; and (3) organizational factors affecting evaluablllty. 



Technical Features Affecting Evaluability 

Some of the most obvious barriers to doing solid, useful evaluation of 
JTPA programs are largely technical In nature. In later chapters, 
several of these technical Issues concerning funding, staffing and 
managing JTPA evaluation efforts will be presented In greater detail. 
For clarity's sake, these technical concerns, as they touch on program 
evaluablllty, are briefly mentioned as follows: 

• Financial Constraints : Are there sufficient funds to ensure 
the evaluation effort's successful completion? If not, can 
additional funds be obtained within an acceptable timeframe? A 
scaled down, but well- supported evaluation effort, providing 
quality Information In a few key areas may prove to be the most 
useful Interim option. (Chapter 9 discusses JTPA evaluation 
funding strategies further.) 



See, for example, Leonard Rutman, Planning Useful Evaluations: 
Evaluablll ty Assessment , Sage Library of Social Research, Vol. 96 
(Severely Hills, CA: Sage Publications, 1980); Joseph S. Wholey, 
Evaluation: Promise and Performance (Washington, D.C.: The Urban 
Institute, 1979); and Richard E. Schmidt, John Scanlon, and James 
B. Bell, Evaluablllty Assessment (Rockvllle, HD: Project SHARE. 
DHEW no.: 05-76-730, 1979). 
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!f?nrl In-house staffing of an evaluation 

effort is one way to overcome financial constraints, but If 
staff resources are stretched thin, this strategy may end up 
compromising, the quality and usefulness of the evaluation. 
Alternatively, such constraints may encourage "creative 
leveraging of both governmental and community resources 
heretofore untapped. (Overcoming staffing constralrrts Is 
treated separately In Chapter 10.) 

fwi^^V'^' ^^'"^''''f'"'^- To be most useful, evaluation must be 
timely In answering the questions of chief Interest to JTPA 
program administrators and policy-makers. If the timeframe for 
collecting and analyzing data Is too liberal, evaluation 
findings may become stale and less relevant to decision-makers. 

M|a_f:ol lection Problems: Insufficient data or Inaccessible 
data n>ay also delimit the nature and scope of an evaluation 
effort. (JTPA data collection Issues are detailed In Chapter 
8. ) 

Program Features Affecting Evaluability 

Another set of factors affecting evaluability has to do with the 
contours of the program Itself, while there may be no substantial 
technical barriers to conducting an evaluation, an employment and 
training program itself may exhibit certain characteristics which make 
evaluation outcomes more difficult to Interpret and utilize 
effectively. Typically, a process (or Implementation) study may be 
necessary to elucidate such features before larger-scale outcome 
evaluations are considered. Some of the characteristics affecting 
evaluability are as follows: 

Changing or Unfocused Rn;^U; Explicit program goals provide a 
predetermined standard against which program processes and 
accomplishments can be measured. When an employment and 
training program's goals are unfocused or constantly changing, 
the task of evaluation Is more difficult: how do you measure 
your achievements If you are not clear about what It Is you are 
trying to achieve? 

* '^"H^P^!, ^"^ — Conflicting RnaU- Program goals may be 
we.l-deflned, but Inconsistent with each other, complicating 
the task of evaluation. For example, the goal of achieving a 
h gh placement rate at a low cost per placement often conflicts 
with other goals, such as significantly developing skill levels 
of participants or long-term retention of trainees In their 
placements. Such goal conflicts are Inherent to many JlPA 
programs; the Issue Is not that of completely ellmlnatino such 
conflicts (an Impossible task!), but making the evaluation 



approach as sensitive as possible to such constraints on 
program outcomes. (A process evaluation may be needed to sort 
out how different program activities are supporting conflicting 
goals 0 

Variable Service Provision Strategies : When programs encompass 
numerous service provision strategies (as is the case in many 
JTPA program settings) or change strategies mid-stream, the 
task of evaluating becomes more challenging. The less uniform 
the overall treatments given, the more complicated the task of 
adequately accounting for program impacts. 

Small Program Size : The size of the program may also shape the 
nature and scope of evaluation. In the case of smaller, 
special JTPA projects or programs (for dislocated workers, 
older workers, etc.) impact findings may be of limited 
usefulness due to small sample problems or cost 
inefficiencies. (A fuller discussion of sample size 
requirements is found in the gross and net impact designs 
presented in Volumes IV and V.) 



Organizational Factors Affecting Evalucioility 

Organizational factors often present the least tangible, but most 

powerful barriers to useful evaluation. Because of the central 

influence they have over evaluation activities, organizational concerns 

will be examined in greater detail in the following chapter. Some 

common organizational factors impacting evaluability are encapsulated 

below: 



Staffing Problems : When a program is plagued with low staff 
morale or high turnover, something is clearly wrong, but an 
evaluation may not help. Evaluation activities may create 
added burdens for the staff which they cannot handle. 
Effective staff are crucial in the operation of any social 
service program. An organization with serious staff problems 
will probably first need to focus its energy on rectifying 
those problems before being able to utilize broad evaluation 
findings. 

The History of Previous Evaluation : Have previous evaluations 
been done? If so, how have they been used? If the results 
have been ignored, is there any evidence to suggest that a new 
evaluation will receive any better reception? Alternatively, 
have evaluations been used to punish or undermine certain 
factions or personnel within the organization? If so, the 
credibility and usefulness of the new evaluation may be 
questioned and staff cooperation lost. Evaluation planners 
will have to develop some initial strategies to build up trust 
and credibility. 
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Hidden Aqendas ; In some cases, the sponsor of the evaluation 
iLr- ]^ comm tted to an open inquiry into program 

operations from which the program can learn or Improve, 
instead the sponsor wants to use the evaluation to support a 
Sorthwhllr'' program as worthwhile or not 

• Financial Difficulties : When a program Is struggling to stay 
afloat Financially, the utility of an evaluation is often 
severely curtailed. Administrative energy is necessarily 
focused on program survival rather than program improvement. 
The program may be able to take better advantage of evaluation 
findings when it is on a more stable financial footing. 

Inter- a nd Intra-Organlzational Relat^nn<: - jurf battles over 
Clients, staff and other resources can compromise the 
evaluation effort. If for example, cooperative support among 
agencies is lacking, the evaluator may find access to important 
sources of information curtailed or delayed in ways that 
negatively impact the evaluation. A comprehensive evaluation 
planning effort will include strategies to ameliorate or 
compensate for difficult organizational relations. 

Improving Program Evaluability 

Some program attributes may impinge upon JTPA evaluation planning in 
ways that are difficult and/or costly to remediate right away. For 
example, the data limitations imposed by a particular MIS may be fairly 
rigid and uncompromising for evaluation plans in the near term. 
However, other limiting factors may be more amenable to change in favor 
of immediate evaluation needs. 

An evaluability assessment is not intended to act as a discouragement 
to evaluation. Part of the assessment task is to help program 
operators determine what evaluability factors can be manipulated to 
enhance overall evaluability. Once those evaluability factors subject 
to influence are identified, evaluation staff can actively work with 
program administration and staff to create a program environment that 
is more receptive to evaluation, staff can tackle not only technical 
evaluability factors such cs data collection levels or methods, but 
also organizational factors such as program goal definition and 
interagency communication. Thus, the benefits of evaluability 
assessment extend beyond preparation for useful, feasible evaluations. 
An assessment encourages program examination and improvements important 
in their own right, apart from any evaluation activity to follow. 
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TO WHAT EXTENT WILL THE EVALUATION BE UTILIZED? 
AS the field of evaluation research develops, there is increasing 
concern over making such research more immediately useful to 
practitioners in the field. This concern is particularly underscored 
in the context of JTPA where limited administrative funds are available 
for evaluation activities. The feasibility of an evaluation and its 
usefulness are obviously intertwined, as the previous discussion on 
evaluability implies. The focus in this section is on increasing the 
usefulness of an evaluation, especially in terms of increasing the 
chances of its utilization. (Utility and utilization are not the same 
thing: an evaluation's results may be useful , in the abstract, but 
still not used.) The rest of this section looks at barriers to 
utilization of evaluation and presents suggestions for minimizing these 
barriers. 



Barriers to Utilization 

The previous section touched primarily on potential barriers to 
planning, conducting and interpreting the results of an evaluation. An 
evaluability assessment is also important in uncovering potential 
barriers to utilization, particularly barriers associated with 
organizational features of a program. What are some of these 
organizational barriers? While a fuller discussion of this question is 
reserved for the next chapter, the following outline provides a glimpse 
of common barriers to utilization. 
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UTILIZATION OF EVALUATION RESULTS: 
SOME POTENTIAL BARRIERS 



• Organizational inertia and resistance to change. 

• Miscommunication between evaluators, those within the 
program being evaluated, and other potential users of the 
evaluation results. 



• Misunderstandings about the purposes of an evaluation. 

• Lack of organizational involvement In or commitment to the 
evaluation process. 

• Failure to sufficiently connect the evaluation to other 
planning efforts. 

• Overly lengthy timeframe for accomplishing evaluation. 

• Unresolved tensions or conflicts between different 
organizational levels or branches of a program. 

• Evaluation team perceived as lacking independence and 
neutrality. 

• Evaluators lack credibility. 

• Evaluation findings not clearly presented or adequately 
disseminated. 



Increasing Evaluation's utility and Utilization 
Many evaluators are now playing a more activist role in ensuring the 
utilization of their findings by program administrators and others. 
Such a role demands that the evaluation group communicate and work more 
in concert with users from the earliest stages of evaluability 
assessment to the issuing of a final report. The following ten points 
summarize the kinds of steps an evaluator can take to build utility 
into the evaluation process from the very beginning. 

1 • Identify "stake-holders" and users of the evaluation . 

g " 

The term "stake- holder** refers to anyone who has a 



The term "stakeholder" is taken from Carol Weiss, "Measuring the 
Likelihood of Influencing Decisions" in Evaluation Research 
Methods: A Basic Guide . Leonard Rutman, ed. (Beverly Hills, CA: 
Sage Publications, 1980. pp. 159-190. 



stake in the evaluation process and its results. Stake holders 
can include program funders, administrators, planners, 
policy-makers, front-line staff, clients and client-advocates. 
While not all of these groups may be directly involved in the 
evaluation process, it's important to know who these parties 
are and how their interests or concerns might affect the 
evaluation and its utility. 

Involve stake-holders 1n the planning process . 
Where feasible, stake-holders who are potential users of the 
evaluation results need to be involved early on in the 
evaluation planning process (starting with evaluability 
assessment) for at least two major reasons. First, potential 
users have to be committed to the particular evaluation chosen 
and believe in its utility to JTPA program improvement. 
Participation in the planning process helps to build user 
understanding of and commitment to the evaluation effort. 
Second, user input helps focus the evaluation on the legitimate 
concerns and interests of the various users. The evaluation is 
more likely to produce information that critical actors in the 
program will want to use, as opposed to information that is of 
interest only to the evaluation staff. 

Educate potential users . 

One problem in basing evaluation around user input is that 
users* initial focus may be restricted to issues of immediate 
programmatic concern. For example, users at the SDA level may 
primarily be concerned with compliance and monitoring issues. 
While these concerns need to be addressed, evaluation planning 
can provide users the opportunity to explore broader evaluation 
options . 

Focus evaluation on users' key question s. 

Ultimately, the evaluation must yield" information that users 
feel will be important to them in answering questions about 
JTPA programs. Achieving such a focus is not always easy 
because different users will bring to the planning process 
different perspectives as to what information is most useful 
and important to obtain. 

Maintain neutrality and Impartiality . 

To be useful , an evaluation must be credible to JTPA 
decision-makers and others. In large measure, such credibility 
rests with the independence and neutrality of the evaluation 
staff. Positioning of the evaluation staff within an 
organization, and the relationship of that staff to JTPA 
administrators and policy-makers are important factors 
influencing the perceived or actual independence of that staff. 

Develop mechanisms for Interim feedback . 

The evaluation process is often a lengthy one. Where possible, 
interim reports, newsletters or presentations help sustain 
users* interest and commitment to the evaluation. 
Traditionally, evaluators have avoided such information 
exchange with users for fear that such feedback might 



35 

43 



contaminate data. However, more recently, commentators have 
suggested that such fears have been overstressed and need to be 
balanced against the practical advantages such Interim feedback 
offers to practitioners. 

Develop a dissemination strategy . 

Traditionally, dissemination Is almost an afterthought to an 
evaluation plan, Involving little more than sending copies of 
the final report to the evaluation funders or perhaps seeking 
publication of the findings In an academic journal. Expanding 
the usefulness of an evaluation, however, calls for a broader, 
more creative approach to dissemination. Such a broader 
approach might Involve: ^ 

• Targeting Important users and other Interested parties 
ahead of time and maintaining contact with these groups via 
newsletter or Interim reports. 

i 

• Planning In-person presentations to various users to allow 
for direct questions and answers about evaluation findings. 

• Where appropriate. Identifying other opportunities to 
present findings to a larger forum of practitioners, as 
well as researchers, such as a conference or special 
publication. 

• Discussing ahead of time how users might be Involved In 
dissemination and whether users will be given formal credit 
or recognition when findings are presented. 

• Considering In what manner public affairs staff might 
assist In presenting findings. 

Produce a clear, well-written report of findings . 
A lengthy, jargon-filled report emphasizing the technical 
aspects of an evaluation creates what Weiss terms ••cognitive 
obstacles*' to Its utilization. Utility of an evaluation 
obviously Increases If findings are pitched to a broad audience 
of Interested parties. Ways to Increase readability Include: 

• Presenting a separate executive summary of findings which 
highlights the most Important conclusions. 

• Placing technical Information, where possible. In a 
separate chapter, appendix or In footnotes. 

• Prominently featuring, through formatting and placement, 
the main evaluation questions. Interpretations, findings 
and recommendations. 

• Adding a glossary of technical terms. If necessary. 

• Packaging evaluation findings differently for different 
audiences. For example, pairing technical summaries of 
findings relating more to research Issues or conclusions 
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with more "user friendly" summaries relating to policy 
Issues of current Interest to decUlon-makers • 



9. Present findings In a timely fashion ^ 

Timing Is all Important In the reporting of evaluation 
findings. If too much time has elapsed between evaluation 
planning and reporting/ the Information presented may no longer 
be fresh or relevant to users. On the other hand, the user's 
call for timeliness must be judged against the need to acquire 
reliable and val id information through acceptable research 
procedures, all of which takes time. The point is not to rush 
through with dubious results, but to agree upon a responsible 
timeframe initially and then stick to it, 

10, Iiffced evaluation 1n ongoing planning cycles . 

Ideally, evaluation plays an integral role in an organization's 
overall planning processes. Evaluation provides feedback at 
critical junctures in a program cycle, allowing planners and 
policy-makers to make informed judgments about the future 
direction of the program. If evaluation is simply tacked on to 
3TPA programs as an afterthought and is not coordinated with 
other .l^PA planning efforts, then evaluation's utility is likely 
to be diminished, 

CONCLUSION 

This chapter is meant to encourage JTPA evaluation planners in the hard 
exploratory planning work that establishes a solid foundation for later 
evaluation activities. Scrutinizing a program for evaluability may 
sound like unnecessarily discouraging or time-consuming work. However, 
discovering potential program constraints to evaluation early on wil' 
give evaluation planners an edge in introducing feasible evaluation 
activities in a more effective manner. 



A related concern is whether the evaluation findings will be 
sufficiently utilized to justify the evaluation effort. In order to 
ensure ultimate use, planners need to anticipate potential barriers to 
utilization and actively build Into the evaluation plan strategies for 
increasing the usefulness and utilization of findings. Since the 
organizational context Informs and shapes the evaluation strategy in 
Important ways, the following chapter looks more closely at the 
relationship between organization and evaluation. 
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CHAPTER 4 

UNDERSTANDING THE ORGANIZATIONAL CONTEXT 

How Will the Organizational Context Impact Evaluation? 

Overcoming Organizational Inertia 
The Evaluation and the Evaluated 
Reducing the Threat of Evaluation 

Who Will Participate In and Support Evaluation? 

Who Should Do the Evaluation? 
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CHAPTER 4. 

UNDERSTANDING THE ORGANIZATIONAL CONTEXT 



INTRODUCTION 

Evaluation does not occur In a social vacuum. Just as political and 
organizational factors Influence JTPA program design and operation, so 
will such factors Influence the nature and scope of evaluation. The 
JTPA organizational context Is complex, cross-cutting all levels of 
government and embracing numerous agencies and organizational agendas. 
Because of this complexity, understanding how organizational factors 
might Intervene to help or hinder evaluation Is especially critical to 
the JTPA evaluation planning process. For example. In such a 
multi-layered program as JTPA, various organizational tensions and 
conflicts are bound to occur. The system may not have mechanisms to 
respond and the conflicts can spill over Into the evaluation process. 

In addition, the evaluation Itself may subtly In-luence program 
processes and outcomes. Therefore, not only the ntext In which 
evaluation occurs, but also the manner In which evaluation Is carried 
out (In Interaction with the context) Is also Importa t. For these 
reasons, preliminary planning for evaluation mu?;t Include a focus on a 
third preliminary planning question: How does the r -qanlzatlonal 
IfDBaci evaluation? When Ignored o'ur .n, ths ev? ijatlon planning 
stage, underlying organizational conflicts c ..■ «. upt, creating 
tremendous roadblocks to later Implementation and utilization of 
evaluation. Related questions are: Who will participate In and 
support evaluation? Who should do the evaluation? The purpose of this 
chapter Is to more fully explore these questions about the 
organizational context of a JTPA evaluation at the local level. 
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HOW WILL THE ORGANIZATIONAL CONTEXT IMPACT EVALUATION? 

For the evaluatlor. planner, the challenge 1s to identify and 

knowledgeably work with organizational constraints and supports to 

evaluation. Since these constraints and supports will vary from 

program to program, the Intention here is to provide a general 

framework for Incorporating organizational Issues Into the evaluation 

plan. 

Overcoming Organizational inertia 

To accomplish their specified missions, organizations create structures 
to promote stability and efficiency. Organizations develop structures 
which establish chains of authority and accountability, standardize 
operations, and routinize and parcel out work in a specific manner. In 
creating stable structures, organizations also create vested Interests; 
a major goal of the organization becomes self-preservation. Over time, 
the ve^y structures developed to enhance the organization's efficient 
functioning have a tendency to become rigidified and resistant to 
change. Change means more uncertainty and, as such, constitutes a 
threat to the organization and its vested Interests, 

The logic of evaluation, on the other hand, is based on the potential 
for change . Ideally, evaluation feedback offers a rational mechanism 
for planned change in the Interest of program improvement. Therefore, 
as a harbinger of such change, the evaluator can expect to encounter 
sone natural organizational resistance to evaluation activities. 
Sometimes the resistance is not active, but takes the form o'r passive 
Inability to mobilize for an evaluation effort. Sheer organizational 
Inertia— the urge to follow time-honored structure and patterns which 
have shaped the orgamlzatlon's identi ty--inh1b1ts the evaluation 
undertaking. 

Some might suggest that ^ h^ce JTPA is relatively fresh legislation, its 
programs have not yet had the time to solidify and build up an 
organizational inertia. However, while JTPA legislation is new, in 
many cases the local program structures and personnel uti lized to 
Implement it are not. Program continuity from CETA days has 
undoubtedly helped many states and SDAs to mobilize for a JTPA effort 
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more effectively. By the same token, program continuity means that 
many JTPA organizations are actually long-established with well-defined 
Interests and are likely to resist evaluation geared towards program 
change. On the other hand. In an age of shrinking public resources, 
JTPA and other programs are under constant external pressure to Improve 
(1.e«, be more productive with fewer resources). Evaluation provides a 
tool for such change which need not threaten the security and 
continuity of the organization. 

Overcoming organizational Inertia or outright resistance to evaluation 
may present more of a challenge than the actual evaluation Itself. 
JTPA's complex administrative structure may demand that not one, but 
several separate organizational entitles be mobilized to cooperate and 
participate In evaluation activities. If those activities are to be 
meaningful . 

To accomplish this mobilization, evaluation planners may have to 
broaden their traditional role to Include education, mediation, 
communication, and public relations activities preparatory to planning 
the evaluation Itself. A common organizational fear Is that the 
evaluation results will only point out program weaknesses and damage 
program credibility. Program administrators and service providers need 
to be assured that the evaluation results can enhance program 
credlliinty In several ways: The fact that a program embraces 
evaluation as a tool for Innovation and Improvement Itself sends a 
positive message to program sponsors. Moreover, a balanced program 
evaluation will help Identify program strengths, as well as weaknesses, 
uncovering program accomplishments which compliance measures do not 
take Into account. And finally, evaluation may produce Information 
that compensates for or explains lower compliance with performance 
standards. 

The Evaluator and the Evaluated 

Even If only temporarily, the evaluator also becomes a part of the 
organizational landscape In which he or she Is operating. How those 
being evaluated perceive the evaluator and how the evaluator. In turn. 
Interacts with those he or she observes, must Inescapably Influence the 
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evaluation process. For these reasons, the evaluator must be sensitive 
to his or her role as an Innovator within the organization and 
anticipate potential difficulties arising from that role. The first 
big challenge for the evaluator Is to reduce the threatening aspects of 
this role. 

Regardless of the specific purpose behind an evaluation, the evaluator 
wishes to be regarded as a facilitator of positive change within the 
system being evaluated. However, It Is difficult for those being 
evaluated to embrace the evaluator' s most positive point of view: 
their natural prejudice Is that the evaluator has come to point a 
disapproving finger at what they are doing wrong. If nothing Is done 
to soften this negative predisposition to the evaluator. If no 
assurances and protection are given to the evaluated, then an 
evaluator's presence Is likely to Induce a defensive posture that Is 
not conducive to the ultimate goals of the evaluation. 

If program staff feel unsure of the purposes behind the evaluation, 
their defensive actions can seriously undermine the evaluation 
process. For example. In one case, JTPA evaluators were Investigating 
the Impacts of a special JTPA program through use of a comparison group 
of non-participants. When the evaluation was In progress, the 
evaluators discovered that program staff. In their eagerness to prove 
the program's worth, became unofficial program gatekeepers- assigning 
for JTPA services only the most obviously job-ready. As a result. It 
became difficult to assess whether positive outcomes were due to the 
program services or to the select nature of clients receiving those 
services. 

The evaluator unavoidably has an affect not only on the social climate 
of a program (an Intruder on sacred soil) but also on the working 
conditions within the program. In requiring Interviews and planning 
meetings, the evaluator distracts staff and administrators from their 
regular work load. Whether staff perceive evaluation duties as a 
burden or an Intrusion depends. In part, on the sensitivity of the 
evaluator and how well staff are briefed as to the nature of the 
evaluation and the Importance of their role In the evaluation process. 
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In a positive context, evaluation interviews and planning meetings can 
offer SDA and service provider staff a chance to be heard and make a 
meaningful contribution. In addition, an evaluation project generates 
its own phoning, typing and other office requirements, which may place 
extra burdens on an already overloaded support staff. Resentments over 
this new work can build if expectations for program staff participation 
are not initially clarified with the evaluation staff. 

The evaluator's (or evaluation staff's) perceived status may also be 
significant to the success of the evaluation. If, for example, the 
evaluator is perceived to be too closely aligned with the administra- 
tive power structure, this perception may impair the credibility of the 
evaluator and his or her ability to carry out evaluation functions. On 
the other hand, if the evaluator is perceived as lacking sufficient 
administrative support, he or she may be seen as "marginal" in relation 
to ongoing program operations. The message is that evaluation is not 
really valued and participant cooperation in the effort may be 
undermined. 

Finally, the evaluator must confront the possibility that his or her 
presence constitutes an additional intervention, or independent 
influence which may affect the program in an unknown fashion. If, for 
example, the evaluator is seen as a threatening presence, staff morale 
and program effectiveness may decline. Alternatively, staff may take 
extraordinary measures which artificially and temporarily boost program 
performance. Even if the evaluator is viewed in a strictly neutral 
light, the subjects of the evaluation (who may range from JTPA clients 
to PIC members) may simply react to the process of being studied (the 
well-known Hawthorne effect). 

While such Influences cannot be totally eliminated, evaluators, in 
being sensitive to their role within the organizational setting, can 
seek to minimize their impact on the research process. The sample 
checklist which follows on the next page summarizes how the evaluator's 
role can be clarified, not only to help ,the evaluator but also the 
program staff, administrators and evaluation sponsors who must interact 
with the evaluator. 
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CLARIFYING THE EVALUATOR ' S ROLE: 



YES NO 

1 I I I 1- Evaluator-Adminlstrator Authority Is there 

written agreement about the evaluation 
decisions and who will be Involi'ed In making 
those decisions? 

I_l l_l 2- Evaluator^-Adminlstrator Responsibilities Have 

responsibilities for both administrators and 
evaluators been clearly defined In writing? 

I I I I 3. Evaluator-Staff Responsibilities Has the 

degree of program staff participation and work 
responsibilities been defined and put In 
writing? 

I I I I 4. Communications Have formal channels of 

communication among the various evaluation 
participants been established? 

I I I I 5. Resources Utilization Are there written 

agreements about the use of In-house resources 
(e.g. , phones, copying equipment, office 
space, etc.) by the evaluation staff? 

I I I I 6- Disagreements Are there written procedures 

for resolving disagreements between program 
and evaluation staff when they arise? 

I„l l_l 7. Briefing Staff Have program staff been 

briefed on the above relevant agreements? 

I I ! I 8. Involving Staff Are opportunities for 

Interaction and exchange of Information with 
program staff scheduled Into the evaluation 
process? 

I I I I 9. Introducing the Evaluators and the Evaluation 

Has Initial time been set aside for Intro- 
ducing the evaluator and evaluation plans to 
the staff and allowing for staff guestlons. 

I I I I 10. Evaluator Influence on the Program Have the 

evaluator 's planned activities been assessed 
for possible Influence on program operations 
and outcomes? 



Adapted from: Kay Adams and Jerry Walker, Improving the 
Accountabllliv of Career Education Programs: Evaluation Outlines 
and Checklists , the National Center for Research In Vocational 
Education^ Crlumbus, 0H» 1979 
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Reducing the Threat of Evaluation 

The evaluator 1s not automatically doomed to alien status within a 
hostile and mistrustful program environment. Although some 
organizational factors may be beyond the evaluator 's control, the 
evaluation plan can Include several strategies to demystify evaluation 
and reduce a program staff's Initial fears about the evaluator and the 
evaluation process: 



• Involve not only program administration, but program staff, as 
well. In the Initial and subsequent evaluation planning 
activities In order to enhance user understanding and commitment 
to the evaluation. 

• Make clear to program personnel the purposes and anticipated 
consequences of the evaluation. Ideally, consequences center 
around constructive program change so that the program can be 
allowed room to fail, but then move on. Remove the threat of 
sanctions being attached to the^ evaluation to the degree 
possible. 

• Emphasize the evaluation of programs , not personnel . The more 
emphasis placed on evaluating the program attributes, as opposed 
to staff attributes, the less threatening the evaluation 
process. If staff Inadequacies are a central concern, then 
other vehicles besides program evaluation should be considered 
to address this concern. 

• Establish clear lines of authority separating evaluation staff 
from program administration staff. 

• Introduce an Initial evaluation effort Into the least 
threatening program situation. For example, focus Initial 
Inquiry on overall program structures, processes or outcomes, 
rather than on Individual service providers 

• Assure confidentiality to clients, staff and all other 
participants In evaluation. 

• Select evaluators whose organizational status 1s perceived as 
most neutral and non-threatening. 



WHO W7LL PARTICIPATE IN AND SUPPORT EVALUATION EFFORTS? 
Numerous distinct state and local level organizations are Involved In 
the administration, planning and Implementation of JTPA activities. At 
the local level, PIC members, SDA officials and staff, city planners 
apd policy-makers, and service providers are, all participating to 
varying degrees In JTPA. In addition, city council members, business 
groups, local welfare and Employment Service offices md the local 
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economic development agency may play an active or Influential role. 
All these organizational actors have developed a stake within the JTPA 
system and therefore have a legitimate Interest In evaluation design, 
Implementation and outcomes which affect them. 

Before launching into a full-blown evaluation effort, one should 
consider the roles these various organizational actors play within JTPA 
and how supportive of evaluation they are likely to be. How active or 
central a role does each organization play? How receptive to or 
constrained by evaluation are key actors? What explicit or implicit 
agency agendas might affect the evaluation effort? Ignoring the 
Interests of a particular JTPA stake-holder in the planning phase may 
Impede the evaluation in later implementation and utilization phases. 

Interorc^anizational Relations: Conflict or Cooperation? 
It is not sufficient to know who the organizational actors are and what 
their stakes in JTPA entail; one also needs to know how these various 
groups Interact with one another. Existing organizational patterns of 
interaction are often best understood in a historical context. Some 
organizational elements of JTPA (like many of the reconstituted PICs) 
are totally new, while others have an Important history predating JTPA 
and influencing current patterns of interagency cooperation, 
communication and coriflict. 

Do the PIC, local program staff, local officials and Involved agencies 
regularly communicate with each other? Are there unresolved turf 
battles ever JTPA or other program areas? Have personality conflicts 
marred Interagency cooperation in the past? These are the kinds of 
questions an evaluation planning group will have to pose and answer in 
order to lay the organizational groundwork to support an evaluation 
effort. 

Conflicting interests 

Sometimes organizational interests are pitted against each other in 
ways that make coordli-s^ted evaluation very difficult. Conflicting 
Interests are most likely to arise where two agencies share the same 
client base, as is the case with many JTPA and welfare programs. 



Competition between these two programs can be particularly Intense when 
the fuller funding of JTPA hai translated Into less funding for welfare 
recipients enrolled In JTPA, and It Is no longer in the Interest of 
welfare agencies to refer clients to JTPA. Nor, for that matter. Is It 
in their Interest to participate in an evaluation which might validate 
JTPA at welfare's expense. 

If agencies have a history of poor communication or turf battles over 
who should administer what programs or who should set policy, this 
history can spill over into and stymie evaluation efforts in Important 
ways : 

• Access to necessary data, program documents or clients may be 
delayed or made more difficult. 

• Otherwise useful in-house resources may not be discovered jnd 
shared. 

• The organizational input necessary for formulating useful 
evaluation questions may not occur. 

• The general utility of evaluation findings may not be recognized 
by important decision-makers. 

These kinds of potential obstacles are especially worth considering if 
a process evaluation is contemplated. Access to various agencies and 
rapport with agency staff will be Important to the evaluator hoping to 
get at key processes and Interactions relevant to the JTPA system. 

Conversely, identifying potentially positive interagency connections 
provides a base on which an evaluation effort can be built. Evaluation 
activities that cross agency or divisional boundaries, while providing 
extra challenges to planning and coordination may also provide unique 
opportunities for the exchange of information and ideas within the 
overall JTPA organization. Since evaluation often requires special 
coordination among different units, the evaluation process can create a 
supportive context for interaction across territorial lines. Such 
interaction can Itself be valuable in informing people about 
decision-making and work agendas in different agencies, reducing 
organizational isolation and improving coordination of resources. 
Whatever the organizational configuration, the planning role cannot 
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remain purely technical. The evaluation planner may need to play 
Information broker and mediator, acting as a conduit to open up or 
enlarge channels of communication and cooperation. 

Cooperative Evaluation Planning 

If JTPA evaluation Is not to be loft In a thicket of organizational 
agendas. It Is iRiportaul that central actors are able to jointly 
participate In planning efforts and arrive at some common understanding 
as to how evaluation Is to benefit JTP/^ as a whole. Building up such 
mu -tl-faceted participation Is ^ challenging task because JTPA concerns 
multiple actors with multiple Interests, needs, and fears, who are 
often not used to working across divisional undar les. 

As stake-holders In the JTPA system, agencies/actors need to feel that 
they are each getting something out of participating In the evaluation 
effort. A crucial task becomes eliciting rrom primary actors what It 
Is they are willing to give and get In return as participants In the 
evaluation process. Also the task Is to help sensitize actors to each 
other's concerns, bringing covert Issues Into the bargaining arena 
(e.g., the perennial problem of dcta acquisition across agencies) so 
that necessary agreements can be negotiated upfront before evaluation 
commences . 

WHO SHOULD DO THE EVALUATION? 

The organizational context should also Influence who plans. Implements 
and administers a JTPA evaluation. Should the employment and training 
staff have primary responsibility for evaluation or should a 
policy-making body like the PIC? Or should an organization more 
removed from the JTPA system have primary evaluation responsibilities? 
Should evaluation responsibilities be divided? Clearly, given enormous 
organizational variation across SDAs , no one organization Is the 
"right" place to house an evaluation effort. What works well In one 
local setting may not be transferable to another. Below are some 
factors In choosing and locating an evaluation staff. 

Authority Structure : The position of an evaluation staff 
within an organizational hierarchy Is Important. Ideally, evaluation 
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staff will be sufficiently detached from the existing hierarchy so that 
they are not perceived to hold any direct power over those being 
evaluated or, conversely, those In a program being evaluated do not 
have direct authority or Influence over the evaluators. Such 
detachment Is often sought by contracting out to a private consultant 
or by establishing an Independent evaluation unit. 

When the head of an evaluation unit reports directly to chief 
declslon^makers In an organization, evaluation activities are more 
likely to be better supported (fiscally and politically) and evaluation 
Information better utilized by managers and policy-makers. Such a 
direct link to power holders, however, may need to be offset with extra 
efforts to bring a range of appropriate division administrators and 
relevant staff into the planning process. Otherwise, there is the 
danger that those lower down will feel compromised by or excluded from 
Important decision-making and become less supportive of the evaluation 
effort. 

When an employment and training agency is attempting its own in-houst 
evaluation, sufficient detachment of evaluation staff may be more 
difficult to achieve. This is not to argue that self-evaluation should 
be abandoned. Rather, the financial and other practical merits of this 
approach need to be weighed against the potential structural drawbacks 
of having a less organizationally autonomous evaluation staff. 

If an in-house evaluation unit is used, a key issue is placement of 
that unit. When the unit is not completely separate from other 
operations , its members may be in the uncomfortable position of 
evaluating JTPA operations managed by people above them in the 
organizational hierarchy. 

Compliance vs , Evaluation : The JTPA authority structure at 
the local level is partially defined by who conducts compliance-related 
activities. Many SDAs have developed special monitoring and compliance 
units which routinely collect and analyze JTPA program data and audit 
certain aspects of JTPA program operations. Since these units are 
already collecting some information about JTPA and since evaluation is 
often viewed as an elegant offshoot of monHoring, the tefrptation is to 



lump evaluation activities In with ongoing monitoring and compliance 
operations. (This tendency 1s probably also reinforced by the CETA 
legacy of mingling compliance and technical functions under one roof.) 

From a purely technical standpoint, piggybacking evaluation onto 
ongoing monitoring operations may make sense: staff are familiar with 
the data and with program operations and personnel. However^ from an 
organizational standpoint, such an arrangement may be quite 
problematic. As mentioned earlier, downplaying the threatening aspects 
of evaluation and enlisting the cooperation of those being evaluated Is 
an Important Ingredient to planning a successful evaluation. The 
neutral, non-threatening posture an evaluation staff seeks Is readily 
compro:?i1sed In the eyes of those being evaluated If that same staff Is 
also connected with compliance activities. The Inherently threatening 
aspects of evaluation are heightened by the fact that the office which 
evaluates Is also the office which critiques and sanctions. A 
compromise approach might be to Involve monitoring and compliance staff 
as special evaluation consultants who can provide unique Information 
and Insights Into JTPA program operations, allowing others to actually 
Implement the evaK.atlon. 

Independence and Neutrality : An evaluation staff's perceived 
neutrality Is closely connected to Its position In the organizational 
hierarchy. If the objectivity of evaluators Is questioned either by 
decisionmakers or those being evaluated, the whole purpose of the 
evaluation effort may be called Into question and the potential utility 
of that effort lost. 

The quest for neutrality does not Inevitably leaO to expensive outside 
consultants. First, hiring outside consultants does not automatically 
remove the suspicion of bias- -outside evaluators may merely be viewed 
as an extension of those who hire them. Second, there are alternative 
approaches to JTPA evaluation that sufficiently meet the requirements 
of Independence and neutrality. For example, evaluation can be 
accompl Ished through an Independent research unit under the PIC, the 
SDA administrative entity or under local government (For a further 
comparison of different evaluation staffing strategies, see Chapter 10.) 



Trust: Trust 1s another important consideration in deciding who 
is best able to carry out an evaluation effort. Trust enhances the 
ability of the evaluator to gain entry to a program and elicit 
information and assistance from program administration and staff. 
Where the relationship between the PIC and SDA staff, for example, is 
characterized by a certain amount of tension or mistrust, a distancing 
of evaluation staff from both organizations may be important. 

An evaluator's neutrality does not necessarily guarantee trust or 
vice-versa. In fact, trust may be based on the evaluator's perceived 
positive bias towards a program. In choosing the evaluation staff, 
trade-offs may have to be made between who has greatest rapport and 
access to program information and who has greatest neutrality and 
independence. 

Competency: Technical competency of an evaluation staff is a 
primary consideration to factor into a decision about how to build an 
evaluation capability. Without proper technical expertise, an 
evaluation is more likely to waste resources a>id produce results of 
questionable validity and usefulness. Technical competency and 
efficiency, while of primary importance, should not be the sole 
criteria for location of an evaluation effort. In addition to 
traditional notions of competency and expertise, familiarity with JTPA 
programs and the ability to maneuver within the system and get things 
done are also important attributes for an evaluation staff. 

Coordin ation Capabilities : The more comprehensive the 
evaluation effort, the greater the need to Involve and coordinate. Who 
is best able to perform vital coordination efforts—to bring Interested 
parties together in critical planning stages, to establish Interagency 
agreements about data and resource sharing, to bridge communication 
gaps when necessary? Here again, some argue that these critical 
non-technical competencies must be obtained by hiring an outside 
consultant whose vision can transcend the narrower perspectives of 
individual JTPA personnel. On the other hand, in-house staff, by 
virtue of their superior knowledge of Interagency history and 
personnel, may also be in a good position to perform such coordination 
functions. 



CONCLUSION: DEVELOPING THE ORGANIZATIONAL CLIMATE TO 
SUPPORT EVALUATION 

This chapter has examined some of the major organizational Issues 
confronting a JTPA evaluation planning effort. In every stage of the 
evaluation process , organizational factors can exert profound 
Influences on that process. If organizational support Is lacking, the 
evaluation effort may flounder and ultimately fall. Traditional 
evaluation planning begins with the assumption that the organizational 
context Is set. In contrast, the assumption presented here Is that 
JTPA evaluation planreing must actively consider the organizational 
environment In which that planning takes place. Planning expands to 
Include not only a preliminary organizational assessment of those 
factors likely to Influence evaluation, but also preliminary strategies 
for building better organizational support for evaluation. Some 
suggested strategies. Implicit In much of the preceding discussion are 
encapsulated as follows. 

• Develop leadership support : The Interest and cooperation of 
relevant program heads end other key administrators Is Important to 
obtain before planning reaches too advanced a stage. 

• Educate key declslon-^makers and their staffs : Decision-makers are 
often unaware of the benefits of evaluation and need first to be 
educated before they will support evaluation. Educational efforts 
m<ght Include circulating policy papers , promoting conference 
attendance, or sharing the results of evaluation activities In other 
states. Essentially, leadership needs to be convinced that 
supporting evaluation, even though results m<ght be less than 
positive. Is a politically responsible position to take. 

• Involve key actors : Preliminary meetings with key actors In the 
evaluation process will help shape an evaluation approach that 
acconnnodates a variety of concerns and does not exacerbate Inter- or 
Intra-agency conflict. Staff as well as administrators need to be 
Included In early planning and/or briefing meetings. 



Identify side-benef I ts of evaluation for different participants : 
In addition to the desired Information evaluation 1s expected to 
provide, users will want to know about particular (often 
unanticipated) side-benefits evaluation might yield. Often these 
side-benef Its are Intangible such as Improved agency coordination or 
more positive Interagency relations. (For more on this theme 
refer back to Chapter 1 and to the discussion on measuring 
evaluation benefits In Chapter 7.) 

Develop advisory groups: To ensure greater understanding of and 
commitment to evaluation, some training programs sponsor evaluation 
advisory groups. Group members not only may Include agency 
representatives, but outside professionals or other citizens to lend 
additional support and credibility to the endeavor. 

Develop I nnovative funding and staffing alternat1v3 s: Sources of 
support for evaluation exist beyond the usual organizational 
channels. Moving outside an agency for evaluation resources can 
extend the base of Interest and support for such activity. (More on 
this point in Chapters 9 and 10.) 

Put Inte ragency agreements and assurances 1n writing ; Successful 
evaluation often depends upon Interagency cooperation and sharing of 
resources. Since control of resources Is always a sensitive 
organizational issue, negotiated agreements about access to data, 
clients, staff and other resources need to be in writing to avoid 
future misunderstanding. 

Use a team planni ng approach ; A team approach to planning makes 
sense where a lot of inter- or Intra-agency coordination and 
communication is necessary for accomplishing evalpjation tasks. Even 
if an outside evaluator Is brought In to do the work, a team might 
also play a useful advisory role, providing a mechanism for more 
direct organizational involvement and commitment to the evaluation. 
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SECTION 3 
TOWARDS A JTPA EVALUATION PUN 



These next chapters continue with an exploration of the evaluation planning 
process. As in the previous chapters, this process is approached through a series 
of key questions confronting the JTPA evaluation planner. In the course of 
answering these questions, the planner follows a roughly sequential set of steps 
culminating in a practical, comprehensive plan for carrying out a JTPA evalua- 
tion effort. 
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CHAPTER 5 

FORMULATING EVALUATION QUESTIONS AND A 
RESEARCH DESIGN 

What Are the important Questions to Be Answered by Evslyatien? 

What Evaluation Approach Makes Sense? 

What Data, Data Collection and Data Analysis Methods Will Be 

Required? 
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CHAPTER 5. 



FORMULATING EVALUATION QUESTIONS 
AND A RESEARCH DESIGN 

INTRODUCTION 

We began the previous chapter with a set of key planning questions 
about program evaluablHtv . utilization of evaluation , and the 
organizational context In which evaluation occurs. We now turn to an 
additional set of planning questions which help to define the nature 
and scope of particular JTPA evaluation activities. These questions 
suggest a general planning sequence culminating In a specific 
evaluation research design. This sequence Is built around the 
following explorations: 

• What are the Important questions to be answered by evaluation? 

• What evaluation approach makes sense? 

• What data, data collection and data analysis methods will be 
required? 

This chapter Is devoted to examining each of these planning questions 
In turn. 

WHAT ARE THE IMPORTANT QUESTIONS TO BE ANSWERED BY 
EVALUATION? 

As has been stressed In the previous section, an evaluation's 
usefulness hinges In large measure on Its providing Information that 
users need In order to make more Informed decisions about JTPA 
programs. The actual design of an evaluation, therefore, develops 
around a kejt set of research questions about JTPA's effectiveness , 
efficiency, or program costs . These key questions will, of course, 
vary at different points In time across different state and local 
program settings, but In general, evaluation will concern one or more 
Q of the following generic questions: 
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TYPICAL PROGRAM EVALUATION QUESTIONS 



• Did the program achieve Us stated goals? 

• Did the program have unintended results 
(good or bad)? 

• Was the program Implemented as planned? 

• How might Implementation be Improved? 

• Who benefited most or least from the 
program? 

• Did program participants as a whole benefit 
significantly? 

• What did the program cost? 

• Which program activities were most/least 
cost effective? 



Defining what are the most significant questions to be answered about 
JTPA will help set the parameters of an evaluation effort early on In 
the planning process. 

Developing Evaluation Questions 

Ideally, evaluation questions are generated by the potential users of 
the evaluation (also referred to as "stake-holders"). Users are most 
often program administrators, policy-makers or special funders of a 
program; users can also more broadly include other stake-holders such 
as staff and the Interested public. As mentioned previously, user 
participation can be crucial in evaluation planning: user input not 
only increases the user's commitment to the evaluation effort, but also 
foLjses that effort on relevant Issues. 

During the question formulation stage, however, evaluation staff do not 
have to abdicate to users entirely. Sometimes uncovering specific 
questions is a difficult process; users may have problems developing 
researchable inquiries about the program. Because JTPA is so 
tremendously "performance driven", users may have difficulty moving 
from a compliance and Tnonitoring mode to broader inquiries. In such 



cases, evaluation staff can play an Important educative role In 
ellcltino or reformulating questions from various users. Ultimately, 
however, user Interests have to be central to the evaluation If the 
findings and recommendations are to have an appreciative audience. 

Different Users, Different Questions 

Bringing different users Into the question formulation stage can create 
additional challenges for the evaluator because different users may be 
Interested In entirely different questions. Conflicts may surface 
between different decision-making levels or branches of a JTPA program 
as to what Is truly Important to know about JTPA. For example, at the 
service delivery level, program staff may be more Interested In the 
Impacts JTPA Interventions are having on clients. (Are clients being 
placed effectively?) while PIC members may be more concerned with the 
business community's perceptions and Involvement, administrative users 
may be more Intrigued with studying the cost-effectiveness of JTPA; 
political leaders may be more concerned with justifying public 
expenditures or meeting constituents' perceived needs. 

When state and SDA users are jointly Involved In evaluation, there are 
potentially thornier Issues to resolve as to the focus of the 
evaluation. Since the state can ultimately sanction a poorly 
performing SDA, that SDA must be more directly and unyieldingly 
concerned with performance Issues. State JTPA policy-makers, on the 
other hand, may feel less compelled to examine Immediate performance 
outcomes and focus Instead on more long-term effectiveness measures of 
the program. The question formulation stage Ideally can provide an 
additional opportunity for Information exchange and acconmodatlon 
between these two groups. 

Narrowing the Focus of the Evaluation 

Once users and evaluation staff have generated sufficient evaluation 
questions, these questions need to be prioritized and the scope of the 
evaluation determined, according to the time and resources .allotted. 
Even though they seem Important, some questions may need to be 
eliminated because discovering their answers will prove too time 
consuming or costly. 
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Once users and evaluation staff have generated sufficient evaluation 
questions, these questions need to be prioritized and the scope of the 
evaluation determined, according to the time and resources .allotted. 
Even though they seem important, some questions may need to be 
eliminated because discovering their answers will prove too time 
consuming or costly. 



Attempting to answer too many questions in one evaluation effort is a 

60 66 



Narrowing the evaluation focus to a specific set of questions to be 
answered can be one of the more frustrating and time-consuming steps in 
planning a JTPA program evaluation. The process may call for a 
generous dose of mediation and negotiation among different users. It 
may require the preliminary sketching out of various contingencies 
concerning funding, staffing and data collection and the revising of 
questions to meet these contingencies. This planning time is well 
spent if it yields a manageable set of evaluation questions which 
reflect what users most want to know about JTPA. Th^s set of questions 
forms the heart of the evaluation, informing and directing the research 
efforts that follow. 

WHAT EVALUATION APPROACH MAKES SENSE? 

Once key evaluation questions have been selected, the task is to choose 
a research strategy for answering those questions. The issue at this 
stage of planning is what strategy is most appropriate, given the 
nature of the evaluation questions and given numerous resource 
constraints, such as time, staff e;'.pertise, and data accessibility. 

Evaluation Approaches 

There are several basic evaluation research approaches and numerous 
variations on these approaches. Each approach has its own strengths 
and weaknesses and is appropriate to answering particular kinds of 
questions. This entire series on JTPA evaluation focuses on three main 
types of evaluation: net impact . gross impact and process 
evaluations. For an overview of the Important characteristics of and 
differences between these evaluation types, refer to Chapter 2 of this 
volume and the introductory chapters of Volumes III, IV and V. As a 
quick review, these three evaluation approaches are summarized on the 
following pacje In terms of the sorts of evaluation questions to which 
each approach best responds. 
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EVALUATION APPROACH: NET IMPACT 



General Questions Asked : 

What outcomes can be attributed 
to the program, rather than to 
other Influences? 

What service strategies are most 
effective for which groups of 
clients? 



JTPA-Speclfic Questions : 

Do JTPA clients in general do signifi- 
cantly better In the labor market than 
non-participants with similar profiles? 

What kinds of treatments have a greater 
Impact on client earnings? 



Are multi-strategy program approaches 
more likely to have a greater Impact 
than single strategy programs? 

Do some client groups benefit more 
from JTPA (In terms of Increased 
earnings) th^n other client groups? 



EVALUATION APPROACH: GROSS IMPACT EVALUATION 



General Questions Asked : 

What are the post program 
outcomes for program 
pi rtlclpants? 



How are employers affected 
by the program? 



How do treatment results 
compare to one another? 



jrPA-Speclflc Questions : 

What Is the overall picture of partici- 
pant employment, wages and welfare 
status at three months, six months or 
nine months after termination? 

How does this picture change over time? 

To what degree does JTPA participation 
raise or lower the turnover rate for an 
employer? affect training time? affect 
supervision or hiring? 

Which treatment strategies (e.g., 
long-term vs. short-term, QJT vs. Class- 
room training) have more positive outcomes 
relative to other treatment strategies? 



How may placements for clients 
be characterized? 



Do post-program jobs for JTP"^ clients 
resemble primary or secondary market 
positions? Are pos^*:1ons training-related? 
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EVALUATION APPROACH: PROCESS EVALUATION 



General Questions Asked ; 



JTPA-Spedf^c Questions : 



How is the program being 
Implemented? 



Hnw are 3DA polices being 
formul ;ted and carried out? 



Is program Implementation 
affecting program outcomes? 



How are SDA policies and procedures 
affectlr.a JTPA service delivery? 



Are certain service delivery 
arrangements supporting or in- 
hibiting achievoment of JTPA goals? 



What are the characteristics of 
clients served? 



If the set of k«y question: selected straddles more than one evaluation 
approach, hut comprehensive evaluation is not feasible, something must 
give. An obvious option is to pursue only those questions clustering 
around a s<^gle evaluation approach. However, this approach has its 
drawbacks: eliminating all process-related questions in favor of 
impact related ones, for example, may ultimately narrow ths utility of 
the Impact findings: the richness an(l explanatory' capabilities of 
process findings are sacrificed. 

Alternatively, evaluation planners might contemplate multiple, but 
scaled down evaluation approaches to accommodate the v^; ious questions 
that are of greatest Interest. Both the gross impact and the process 
evaluation designs lend themselves to this kind of flexible application. 

Obviously, a number of factors in the real world will Influence the 
kind of evaluation approach selected: evaluation costs, timeframe foi 
accomplishing the evaluation, data requirements, staff and other 
resource capabilities, and organizational demands. But regardless of 
these various considerations, the approach should be driven by 
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questions of central 1niportai»ce to users and funders 1^ the evaluation 
findings are to benefit and be of use. 

WHAT DATA. DATA COLLECTION AND DATA ANALYSIS 
METHODS WILL BE REQUIRED? 

Settling upon a basic evaluation approach Is only the first step In a 
series of research planning decisions about how the evaluation Is 
actually to be carried out--the specific JTPA variables to be studied, 
the kinds of data to be collected, and the manner In which the Cata are 
to be collected and analyzed. The end result of these decisions Is a 
feasible research design for answering the questions Initially posed 
about the efficiency or effectiveness of JTPA, 

Here again, real world considerations Imnlnge upon the choices 
evaluation planners would Ideally like to make. The full range of data 
desired may be too costly or time-consuming to collect In Its 
entirety. Some Information may be difficult to retrieve or 
Inaccessible, Staff may lack expertise In specific kinds of 
statistical analysis required by a research approach. In recognition 
of these kinds of issues, the specific analysis guides (Volumes III, 
IV, and V) for process, gross Impact and net Impact evaluations 
attempt to balance the need for practical, flexible assessment tools 
with requirements for scientific soundness In the research methods 
used. Specific questions about the kinds of data to be collected and 
analysed are addressed In each of these analysis guides. 

Some general data collection and MIS- related Issues cross-rut the 
various evaluation approaches. Is the requisite data available through 
the current information system? Is the data comparable across SDAs? 
What kind of data sharing agreements across agencies will be 
necessary? These sorts of issues will be covered in .nore detaM in 
Chapter 8. They are only mentioned in passing here to emphasize that 
planning for data collection and analysis may involve some special 
challenges to be discovered and met well in advance of evaluation 
implementation. 
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CONCLUSION 

The evaluation design process begins with a set of well-defined 
questions reflecting what administrators, funders or other users most 
want to know about JTPA programs. These questions^ In turn, largely 
determine what the overall evaluation approaches will be. The task of 
the evaluation planner at this stage Is to translate the general 
framework of evaluation questions Into a specific research design for 
accomplishing the evaluation. This task Is the central focus of the 
net Impact, gross Impact, and process evaluation guides In this 
series. 

The following chapters assume that planners have already considered the 
Important research design Issues and are now able to move on to more 
specific resource and implementation planning. 
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CHAPTER 6 
DEVELOPiNG AN EVALUATiOM PUN 

Wha^ Does a Good Evaluation Plan Entail? 
What Resources Will Evaluation Require? 
What Time Schedule Will Evaluation Activities Depend Upon? 
How Will Evaluation Activities Be Monitored? 
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CHAPTER 6. 



DEVELOPING AN EVALUATION PLAN 

INTRODUCTION 

Once the major evaluation research questions outlined In the previous 
chapter have been resolved (a research approach and design selected, 
data collection and analysis Issues resolved), evaluation planners can 
think more specifically about how the evaluation will be implemented 
and can ch^rt a course for the evaluation activities to follow. This 
course of planning Is highlighted by the following questions: 

• What does a good evaluation plan entail? 

• What resources will the evaluation require? 

• What time schedule will evaluation activities depend upon? 

• How will evaluation activities be monitored? 

• What will the evaluation cost, and will potential benefits 
outweigh the cost? 

This chapter will tackle the first four questions . The Issue of 
evaluation costs and benefits will be reserved for the following and 
final chapter of this section. 

WHAT DOES A GOOD EVALUATION PLAN ENTAIL? 

A written e valuation plan Is an Invaluable tool for both 
conceptualizing and candying out well- coordinated, tim-ily, and useful 
evaluation activities. Ideallv, an evaluation plan comprehensively 
documents all the various planr^ln^ and management decisions which must 
precede and direct the actual carrying out of the evaluation. 

Committing this plan to writing I5 helpful In ^»^?veral ways. First, a 
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written plan creates a conceptual record which can continually be 
referred to for clarification and direction. As a written record, the 
plan Is more subject to outside review, critique and revision than Is a 
set of plans carried around In someone's head. A written record also 
allows for a more broadly shared understanding of the evaluation 
process and how the conceptual work of planners will shape that 
process. For the evaluation team, of course, such an understanding Is 
crucial to the efficient coordination of evaluation tasks. Evaluation 
users may also appreciate knowing more about the planning 
considerations Influencing the evaluation, as documented In a good 
evaluation plan. 

The Plan as a Blueprint for Action 

Rather than a single document, the comprehensive evaluation plan can 
consist of a number of Interrelated statements, descriptions, charts 
and checklists. Informal notes, memos and interviews can be supporting 
or supplemental documents to the main plan. 

Whatever written format is used, the core of the plan should provide a 
detailed blueprint of the sequential activities occurring in each phase 
of th'i evaluation. The evaluation process usijally encompasses three 
maj phases: 

• a planning phase; 

• an Implementation phase; and, 

• a reporting and dissemination phase. 

Since the activities each phase includes will vary from one evaluation 
sc^tting t ih^r, no set checklist of activities can app'iy to all 
s1tuatio:.v. The evaluation activities listed ^jn the following page are 
meant to Illustrate the generic categories of activities a plan might 
cover. 
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A SAMPLE LIST OF EVALUATION ACTIVITIES 
COVERED IN AN EVALUATION PLAN 



PHASE I: PLANNING 



(1) Collecting background Information on JTRA programs. Including: 

• Reading and analyzing relevant prograr.j related docunients, past 
reports 

• Preliminary meetings with sponsors and other users of evaluation 

• Preliminary Introduction/briefing with program staff 

• Site vlslt(s) 

• Selecting advisory committee 

(2) Assessing evaluablllty 

• Interviewing key staff regarding technical, organizational, and 
political factors affecting evaluablllty 

• Brief outline of findings and recommendations for proceeding 

• Heeting with and feedback from program administrators 

(3) Formulating questions 

• Review by users and advisory corwnlttee 

• Question and answer session with users (feasibility Issues) 

• Final selection of questions 

(4) Developing an >:^valu2t1on research design 

• Review data to be collected (availability, validity, reliability) 

• Data collection procedures (sampling strategy and Interview 
procedures) 

• Data analysis procedures 

(5) Assigning and briefing staff and developing an overall resource plan 
(5) Developing dissemination strategies 

(7) Reviewing by advisory committee 
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EVALUATION ACTIVITIES, continued 



PHASE II: IMPLEMENTATION 

(1) Briefing all concerned staff 

(2) Field testing Interview Instruments 

(3) Data collection 

• HIS data 

• Interview data 

(4) Data cleaning procedures and other preparation for analysis 

(5) Analyzing data 

(6) Interpreting the results 

PHASE III: DISSEMINATION 

(1) Preparing Interim reports 

(2) Reviewing by users and advisory committee/questions and 
feedback 

(3) Preparing final report and recommendations 

(4) Reviewing (formal) 

(5) Preparing artlcle-length summary of evaluation report 

(6) Scheduling question and answer meetlng(s) and final In-person 
presentation 



77 

71 



In serving as the evaluation's blueprint, the core of the evaluation 
plan covers not only activities , but also the costs , timing , resources 
and management which these activities entail. The part of the 
blueprint which focuses on resource utilization and costs Is sometimes 
called a resource plan, which Is the focus of the next section. (A 
specific example of a resource plan is presented In the next section.) 

Statement of Purpose 

In addition to a blueprint for action, the overall plan should contain 
a statement of the evaluation's purposes and goals and the questions 
the evaluation Intends to address. Such a statement acts as a 
conceptual reference point for the rest of the evaluation plan. At the 
end of the evaluation, the statement of purpose also offers a yardstick 
for measuring the evaluation's accomplishments. Did the evaluation 
effort stick to the original goals? Did It serve the purposes It was 
supposed to serve? Did It answer the questions that were posed? 

Summary 

There Is no simple recipe for creating a good evaluation plan. From 
the preceding discussion, several guidelines may be distilled: 

• The plan should be In writing. 

" The plan should be comprehensive. 

• The plan should Include a blueprint for carrying out all phases 
of the evaluation. 

• The plan should cover all evaluation activities, costs, timing, 
resources and management. 

• The plan should contain a statement of purpose and goals. 

A specific checklist for elements In the evaluation plan Is Included at 
the end of this chapter. First, we look more specifically at some 
resource management aspects of the overall plan. 

WHAT RESOURCES WILL THE EVALUATION REQUIRE? 

Since evaluation needs. Interests and capabilities will vary across 
local settings, so will resources required. A resource plan , a written 
strategy for accomplishing the evaluation. Is an essential tool for 
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effectively planning and managing the evaluation effort. The plan may 

begin as a tentative document subject to all kinds of attacks and 

revisions In the Initial stages of evaluation planning. Before the 

actual evaluation focus (which questions are to be answered) and 

approach (what evaluation design Is appropriate) are delineated, the 

plan must be sketchy. But as certain early decision points are 
reached, the plan takes on greater detail and form. 

Elements of a Resource Plan 

A resource plan can be devised according to a number of formats. 
Whatever format Is chosen, the basic elements of the plan Include: 

• A sequential listing of evaluation tasks to be performed and 
products to be produced 

• A time allotment for each task 

• The staff and other resources needed for each task 

• An estimate of the quantity or amount of resources required 
(number of staff hours , computation or word processing time, 
etc.) 

All of ^he above elements need to be Identified 1n[ wr1t1n_g and combined 
in some easily readable form. A simplified example of an evaluation 
resource plan follows . As this example Illustrates , many JTPA 
evaluators will require some special staff or consultant Input at key 
Junctures. For a look at the special staff skills JTPA evaluation may 
call for and other staffing Issues, see Chapter 10. 
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SAMPLE RESOURCE PLAN FOR EVALUATION 



Activities 



Staff Staff Total Other Special 
Assignments Time Time Staff and 
(days) Stf/Other Resources 



PHASE I: Planning 

(1) Collecting background 
information 



(2) Assessing evalu- 
ablllty 

(1) Formulating 
questions 

(4) Developing a 
design 



(5) Assigning and 
briefing staff/ 
developing resource 
plan 

(6) Developing dissem- 
ination strategies 

(7) Reviewing and feed- 
back by Advisory 
Committee; making 
revisions 



Sanchez 
Johnson 
Heller 

Johnson 



Johnson 
Heller 

Johnson 

Chang 

Miller 



Sanchez 
Johnson 



Sanchez 



Sanchez 
Johnson 



2 

10 
5 



5 
2 

2 
7 
7 



2 
2 



17/3 consultrnt (3 days) 



2/2 
7/ .5 
16/3 



4/0 



1/0 



6/1 



consultant (2 days) 



consultant review 
( .5 day) 

borrow statistician 
from agency X for 
review (1 day); 
consult with pro- 
grammar (2 days) 



advisory 

committee*; consul- 
tant review of plan 
(1 day) 



PHASE II: Implementation 

(1) Briefing affected 
program staff 

(2) Conductifiq field test 
of interview instrument 

(3) Jata collection 



Heller 
Johnson 

Miller 



2 

2 



Ch^ng 
Miller 



10 
6 



4/0 

3/4 
16/14 



field test inter- 
viewers (2 days) 

interviewers (10 
days); computer 
time and operators 
(4 days) 

Advisory committee time is not calculated here because it is an external 
resource which is free to the evaluating agency. 
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on 



PHASE I: Planning 

(1) Collecting background 
information 



(2) Assessing evalu- 
abillty 

(?) Formulating 
questions 

(4) Developing a 
design 



(5) Assigning and 
briefing staff/ 
developing resource 
plan 

(6) Developing dissem- 
ination strategies 

(7) Reviewing and feed-^ 
back by Advisory 
Committee; making 
revisions 



Sanchez 
Johnson 
Heller 

Johnson 



Johnson 
Heller 

Johnson 

Chang 

Miller 



Sanchez 
Johnson 



Sanchez 



Sanchez 
Johnson 



2 

10 
5 



17/3 consultant (3 days) 



5 
2 

2 
7 
7 



2/2 
7/ .5 
16/3 



4/0 



1/0 



6/1 



consultant (2 days) 



consultant review 
( .5 day) 

borrow statistician 
from agency X for 
review (1 day); 
consult with pro- 
grammar (2 days) 



advisory 

committee*; consul- 
tant review of plan 
(1 day) 



PHASE 11: Implementation 

(1) Briefing affected 
program staff. 

(2) Conducting field test 
of interview instrument 

(3) Jata collection 



Heller 
Johnson 

Miller 



Ch^ng 
Miller 



2 
2 



10 
6 



4/0 



3/4 



16/14 



field test inter- 
viewers (2 days) 

interviewers (10 
days); computer 
time and operators 
(4 days) 



Advisory committee time is not calculated here because It is an external 
resource which is free to the evaluating agency. 



74 



SO 



EKLC 



The Utility of a Resource plan 

The resource plan is important and useful to the evaluation effort for 
a number of reasons: 



• A thorough resource plan anticipates all activities and tasks 
comprising the evaluation and the kinds of resources necessary 
for the completion of those tasks. 

• In apportioning out the work to be done, a resource plan can 
suggest a realistic timeframe for accomplishing the evaluation. 

• The plain may encourage comparison of alternative allocations of 
resources . 



• The plan identifies resour .e gaps which may need to be filled by 
outside consultants or others. 

• The plan permits administrators to appropriately plan for and 
coordinate the use of special resources, such as extr^i technical 
expertise which may be difficult to obtain on short notice. 

• The plan acts as an ongoing management tool for tracking and 
coordinating multiple activities. 

WHAT TIME SCHEDULE WILL EVALUATION 
ACTIVITIES DEPEND UPON? 

As with any project work plan, the evaluation resource plan should also 
include a specific schedule for the accomplishment of tasks. The 
scheduling dimension is important to the evaluation effort for a number 
of reasons: 



Evaluation Timing and User Commitment: If not accomplished within a 
specified timeframe, evaluation results can go stale. The 
organizational momer 'n behind evaluation may die and the results, when 
finally produced, ma> no longer be valued or utilized. Over a period 
of time, the potential users of the evaluation may change 
substantially. New users may have less commitment or Interest In ths 
evaluation or may feel more threatened by the information .^valuation 
elicits. For these reasons, user input may inform che scheduling , as 
well as content of the evaluation. 

The Time Schedule as a Ranjjgement Tool: Establishing a timeframe is 
also critical to the day-to-day management of the evaluation. Careful 
planning of the evaluation . ^3dufe, In a-ticipating problem 



areas and resource bottlenecks, will lead to more efflc^-^nt resource 
utilization. A detailed timeframe also acts as a moni i;or^[;cj tool for 
keeping task accomplishment on schedule. However, the tlmbframe Is 
only as good as the component task Information of the resource plan. 
The more sketchy the resource plan, the more difficult to realistically 
allocate time and sequence evaluation activities. The evaluation 
resource plan (Introduced earlier) can be easily expanded to Include 
more specific scheduling Information for managing the evaluation: 



SAMPLE RESOURCE PLAN WITH TIMEFRAME 



Activities 



Dates 



(2) Assessing evalu- 
ablllty 

(3) Formulating 
questions 



2/11 - 2/13 
2/13 - 2/18 



Staff Staff Total 

Assignments Time Time 

(days) Stf/Other 



PHASE I: Planning 

(1) Collecting background 2/1 - 2/10 Sanchez 
In/ormatlon Johnson 

Helle'- 

Johnson 



Johnson 
Heller 



2 

10 
5 



17/3 

2/2 
7/. 5 



Time Schedule as a Coordination Tool : The scheduling of an evalua- 
tion should also mesh with relevant funding, legislative and planning 
timetables. For example, evaluation findings with implications for 
broad policy-making might Ideally be coordinated with the policy time- 
frames of the PIC, economic development agencies, dnd local government. 
Evaluation plans might also be coordinated to Inform alloccitlon 
decisions for ^tate set-aside monies or other state and local adminis- 
trative actions. The Important point In overall scheduling Is to 
seize, wherever possible. Important coordination opportunities with 
other actors within the total JTPA system. Such coordination can only 
enhance the ultimate utility of the evaluation effort. 



HOW WILL EVALUATION ACTIVITIES BE MONITORED? 
In scheduling evaluation activities, planners can build into the 
evaluation process opportunities for review, comment, and revision. 
These opportunities for monitoring significant phases of the evaluation 
can enhance the overall evaluation effort in several ways: 

• Re-'-ew opportunities build flexibility into the evaluation plan, 
allowing For changes and improvements where necessary. 

• Review allows for a lternative decision points to be scheduled 
into the evaluation process rather than forcing a decision 
before adequete information is available. 

• Revipw„ in encouraging the timely discovery and correction of 
research problems or planning gaps may ultimately save time and 
resources . 

• External review by an independent third party can increase the 
user's confidence in and overall credibility of an in-house 
evaluat n. 

Review can be scheduled not only for the early planning phases of the 
evaluation, but also following later phases of implementation and final 
reporting. This kind of more comprehensive review offers insights into 
the strengths and weaknesses of the evaluation upon which 
reccnmendations for future evaluation activities can be based. (For 
more on formal evrnuation review, or audit , see Chapter 10.) This 
chapter concludes with a sample checklist for reviewing an evaluation 
plan. 

Reviewing Evalt.tion Plans. 

In concluding this chapter with a plan review checklist, we come back 
full circle to the initial question posed: what does a good evaluation 
plan entail? The review example below suggests four separate frame 
works for assessing the adequacy of a plan: conceptual . 
organizational . research , and management . 
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COFPONENTS IN AN EVALUATION PLAN 



Instructions ; Rate your evaluation plan by checking the appropriate 
descriptive category for each component of the 
written plan. 



Well Partially Not Not 
Defined Defined Defined Appllcabl 



Conceptual Framewo rk 

1. Statement of purpose(s) | | | | |~l I I 

for the evaluation? 

2. Questions to be addressed? | | | | |~| | [ 

3. Users to be served? | | | | |~| | | 

4. Potential users of the results? | | | | | | | | 

5. Overview of evaluation approach, | | | | |~| | | 

research activities? 

6. Evaluation products expected? | | | | | | | | 



Organizational Framework 



7. Methods for assessing evaluabi 1 Ity? | | | | | | | | 

8. Strategies for Increasing leadership and 

organizational support for evaluation? | | | | | | | | 

9. Organizational factors affecting the 

location of evaluation? | | | | | | | 

a. Authority and compliance factors? | | | | |~| | | 

b. Credibility factors- |~| |~| |~| |~| 

c. Neutrality and Indp endence factors? | | | | |~| | | 

d. Technical and other competency | | | | | | | | 

factors? 

e. Coordination capability factors? | | |~| [~l | 1 



PLAN COMPONENTS. Continued) 



Well Partially Not Not 
Defined Defined Defined Applicable 

10. User Involvement and feedback? |~| | — | | — i i — i 

11. Educational and briefing activities? |~| |~| |~| | — | 

12. Advisory group participation? |~| | | i — i i — i 

13. Community participation, community 

resource utilization? | | | j | — | i — i 

14. Evaluator role and responsibilities? |~| |~| \ — i i — i 

15. Program staff roles and responsibilities? |~| |~| } i i — i 

16. Mechanisms for Interim feedback to 

users and program staff? | | | | | — | j — | 

17. Strategies for enhancing |~| | | i — i i — • 
staff cooperation? — — 

18. Intra- and Interagency agreements for 

data/resource sharing? j j \ i i — i i — i 

19. Dissemination strategy? |~| j | | — | | — j 

20. Confidentiality agreements and staff 

protections? | | j I i — i i — . 

Research Framework 

21. Theoretical basis for research design? |~| |~| | | | — | 

22. Data gathering Instruments? |~| | | | i i — i 

23. Data gathering procedures? | i i i i i i — i 

24. Sampling strategy? | | | j [ — I | — j 

25. Data storage and retrieval procedures 

(Including data merging procedures)? | | | | | | | — j 

26. Procedures for reviewing data 

reliability , validity , comparability ? |_| |~| \~\ \~\ 

21. Data analysis procedures? | | i i i i i — > 

28. Data Interpretation methods? |~| | | |~i i~| 
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PLAN COMPONENTS, Continued) 



Management Franiework 



Me11 Partially Not Not 
Defined Defined Def1?ied Applicable 



29. A dissemination plan for findings? 

30. A plan for Interim reports, briefings? 

31. Sequential list of all evaluation 
planning tasks and activities? 

32. Sequential list of all Implementation 
tasks and activities? 

33. Sequential list of all reporting 
and dissemination activities? 

34. List of all products to be produced? 

35. A timeframe for tasks and products 
completion? 

36. Staff and other resources (facilities/ 
equipment) needed for each task? 

37. Procedures for contracting with a 
consultant? 

38. Who will perform various tasks? 

39. Job qualifications and job descriptions 
for staff? 

40. Review procedures? 

41. Policies and procedures affecting the 
evaluation? 

42. Evaluation costs and benefits? 

43. Budget allocations? 



I 



Adapted from: Kay Adams and Jerry Walker, Improving the Accountability 
of Career Education Programs: Evaluation Guidelines and Checklists . 
Columbus, OH: The National Center for Research 1n Vocational 
Education, 1979, p. 69. 
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CHAPTER 7 

ESTiMATSNG EVALUATION COSTS AND BENEFITS 

What Will an Evaluation Cosf? 
Cs3i Evaluation Benefits Be Measured? 
Will Benefits Outweigh Costs? 
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ESTIBIATING EVAUi'ATTON COSTS AND BENEFITS 
IMTRODUCTION 

These volumes on evaluating JTPA are premised on the notion that 
evaluation Is an Important management tool for decision-makers, and 
offers i.e.)/ benefits In terms of Improved understanding and operation of 
jiPA. While perhaps sasllv accepting this premise In the abstract, 
JTPA declslon-makeo-s will want to know the bottom line in more concrete 
terms before coinmlttlng to ;^valuat1on. How much will evaluation cost, 
and will the purported benefits outweigh the costs? The answers to 
such questions are usually not neat and straightforward: the 
benefit-cost calculation is often very elusive. In this chapter we 
examine briefly some of the Issues associated with estimating the costs 
and benefits of evaluating JTPA. 

WHAT WILL Mi EVALUATION COST? 

As Is the case with any plan, estimating the costs of evaluation is a 
critical step in \he planning process. Funders need a preliminary 
price tag before authorizing an evaluation effort, and as early as 
possible evaluation planners themselves will want to anchor evaluation 
option: to concrete financial realities. The thorough costing of the 
major evaluation components provides a realistic basis for comparing 
evaluatio.i alternatives and assessing the relative merits of different 
data collection and staffing strategies. An estimation of costs and 
b-f fits encourages planners to creatively rethink alternative resource 
and staffing strategies or consider one or more scaled down versions of 
the preliminary evaluation design. 

Evaluation costs will vary tremendously depending on the purpose and 
scale of the evaluation effort, the kinds of resources an organization 
can marshal to do the evaluation, and the existing market cost for 
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external resources, such as consultants. For example, consultant fees 
for an evaluation specialist may range from $100/day to $600/day, or 
more. Personal field Interviews can cost from $100 to $500 per inter 
view, depending on consultant fees and how difficult It Is to locate an 
Interviewee and collect the Information. Sometimes reduced fees or 
In-kind contributions are available, altering the cost framework for 
evaluation substantially (see Chapter 9 on alternative funding options). 

The preliminary resource plan provides a ready format for assessing 
evaluation costs. To the evaluation activities, schedules and resources 
columns Is added an additional column for costs, as excerpted below: 



SAMPLE RESOURCE PLAN WITH COSTS 



Staff Total Other Special 
Time Time Staff and 

(days) St f /Other Resources 



Costs 



Collecting back- 
ground Information 



Assessing evalu 
ability 



2 

10 
5 



5 
2 



17/3 



2/2 



7/1/2 



consultant 
(3 days) 



consultant 
(2 days) 



consultant 
review 
( .5 days) 



Consultant : 
$250/day @ 
3 days = $750 

Consultant : 
$300/day (B 
2 days = $600 

C onsultant : 
$25G/day (B 
.5 days=$12£ 



Activities 



PHASE I: Planning 
(1) 

(2) 



(3) Formulating 
questions 



(4) Developing a 
design 



(5) Assign*;;! and 
briefing staff/ 
developing 
resource plan 



2 16/3 

7 

7 



borrow statistician 
from agency X for 
review (1 day) ; 
consult with pro- 
grammar (2 days) 



Agency statis - 
tician : $30/ 
hr X 8 = $240 
(agency rate) 



2 4/0 Agency pro - 

2 g rammer : $30/ 

hr X 16 = $480 
(acency rate) 



[Note: These hypothetical costs cited are only given as a gp .ral example 
how costs must be linked to specific evaluation activities 1 Isted In 
resource plan. The figures d o not reflect actual costs and should not 
taken as representative of evaluation costs In general.] 



Counting All Evaluation Costs 

The above example of evaluation costs 1s overly simplified In that It 

only lists obvious extra costs such as consultants. A truly effective 

cost assessment must Include aH costs borne by the sponsoring agencies 
1 0 

or agency . not just explicit dollar costs. Where 1n-k1nd resources 
such as staff time, computer time, administrative overhead and 
materials are shifted to an evaluation project, those resources should 
also be fully costed out. In such cases, it may be more convenient and 
meaningful to cost out some resources 1n other than dollar terms, such 
as staff hours to be donated to the evaluation. (Examples of various 
evaluation costs appear on the following page.) 

Less Quantifiable Costs 

Quantifiable costs, such as labor and materials, are only part of the 
total cost equation. These costs must be considered In concert with 
other, less definable costs. Examples of this more elusive category of 
costs might Include the level of anticipated program disruption caused 
by the evaluation or resource losses associated with an inexperienced 
staff. 

Perhaps some of these non-quantifiable costs can only be compared 
across different evaluation strategies in terms of the negative Impacts 
on utilization. Consider the strategy of using in-house staff vs. 
outside consultants. In some cases, Uie former strategy may be much 
cheaper, but the results less credible to Important funders or 
decision-makers. While not measurable, the potential costs of reduced 
credibility and utilization are nonetheless i-.portant to the overall 
cost calculus. The chart on the following page categorizes the various 
potential costs both quantifiable and not-so-quantif iable, associated 
with evaluation. 



Evaluation theorists hold divergent notions as to how costs should 
be calculated. See for example, Scriven, Michael, "Costs in 
Evaluation" in Ihe Costs of Evaluation by Marvin C. Alkin and 
Lewis C. Solmon, eds. (Beverly Hills, CA: Sage Publications. 
1983), pp. 27-44. 
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THE COSTS OF EVALUATION 



Quantifiable Costs 
Direct Costs 



• Travel 

• Evaluation staff salaries/ 

benefits 

• Consultant fees 

• Per diem expenses 

• Telephone and mall 

• Computer time for data 

processing 

• Printing/duplication 

• Published materials 

• Supplies 



Indirect Costs Overhead 

• Facilities and space 

• Equipment rental, use and repair 

• Utilities 

• Administrative time 

Support Services 

• Secretarial/office 

• Accounting 

• Legal (e.g., contracting, client 

confidentiality Issues, data 
use Issues, etc.) 

• Public relations 

• Publishing 



Non-Quantifiable Costs 

Potential Costs to 
Staff and Clients 



c Interagency coordination costs 

« Program disruptions 

• Service inefficiencies 

• Interview time 



General Program- • Credibility problems and costs 

related Costs • Mistakes, Inefficiencies of 

Inexperienced staff 

• Time delays 

• Staff resistance to evaluation 

• Inadequate or Inappropriate 

utilization of evaluation 
results 

• Political costs 
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CAN EVALUATION BENEFITS BE MEASURED? 

The costs of various evaluation strategies are most meaningfully 
Interpreted In the context of comparative benefits to be derived from 
each strategy. However, evaluation benefits are far more resistant to 
comparative calculation than are costs. First, most potential benefits 
of evaluation are more difficult to measure or are intangible. The 
primary benefit of evaluation is better Information about JTPA, but 
whether that Information is well-utilized and leads to program 
Improvements is another question. After-the-fact program Improvements 
may be translated into quantifiable program gains (more clients 
referred, more clients served), but no such calculation can be made 
prior to the evaluation. 
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Second, the potential benefits of evaluation are often long-range and 
difficult to predict, not only in terms of degree of benefit, but also 
in terms of who will benefit. The benefits to be derived from 
evaluating a currently successful program may largely accrue in the 
future to entirely different programs In different local or stat,? 
settings. Finally, evaluation may confer on an organisation secondary 
benefits which are often not considered in the benefit-cost equation 
because they are by-products of the evaluation process rather than 
directly related to the evaluation findings. The following section 
discusses the notion of Indirect benefits further. 

Direct and Indirect Benefits of Evaluation 

Anticipated central benefits of JTPA program evaluation will most often 
relate to better Information leading to future improvements in program 
efficiency and effectiveness . These direct benefits of evaluation are 
explored in some detail in Chapter 1. In addition, the evaluation 
process may lead to certain organizational enhancements, or indirect 
benefits, which are not explicitly connected to JTPA goal 
achievement. For example, evaluation planning may result in better 
inter- and intra-agency communication and/or coordination in areas 
beyond JTPA evaluation. Evaluation implementation may result in an 
enhanced HIS or other data collection improvements. Examples of the 
various potential benefits (both direct and Indirect) to be derived 
from evaluation are summar1l6d on the following page, 
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BENEFITS OF EVALUATIO^.^ 



Direct Benefits to • Improved understanding of JTPA 

JTPA Programs program activities and outcomes- 

• Increased accountability to program 
funders/publlc. 

• Recommendations for Improved program 
efficiency and effectiveness, 

* Information for JTPA planners 
and managers 

* Information for JTPA pollcy- 
* makers 

* Information which complements 
and moves beyond performance 
standards 



Indirect Benefits to • Improved Intra- and Interagency 
JTPA Organizations communication/coordination. 

• New contacts within the research 
and professional communities. 

• Enhanced "partnership" with 
business and professional groups. 

- New funding connections and 
capabilities. 

• Improved capabilities for doing 
future evaluation, Including 
Improved program evaluablllty. 

• Enhanced MIS or other data 
collection systems. 

• Improved data cleaning procedures. 

• Increased political credibility. 



Indirect Benefits to •* Lessons learner^ from one evaluation 
Other Programs and setti:ig appl.ed to other settings. 

Individuals 

• Improved services to the intended 
target groups. 



Indirect Soda's • Increased public awareness of and 
Benefits support for JTPA. 

O 89 " 
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WILL BENEFITS OUTWEIGH COSTS? 

Those who are looking for concrete benefit-cost decision rules for 
doing (or not doing) a JTPA evaluation will remain disappointed. We 
can take some comfort In the fact that cost factors are relatively 
discrete and quantifiable, allowing decision-makers to more readily 
compare costs of competing evaluation alternatives (and, or course, 
competing non-evaluation uses of resources). The difficulties come in 
plugging evaluation benefits into the equation; how can one assign 
measurable value to the various informational and organizational 
benefif a JTPA evaluation can yield? Evaluation clearly does not lend 
Itself to any straightforward balancing of numerical costs and benefit? 
to see which outweighs the other. 

However, the Inability to assign costs and benefits along the same 
quantitative dimensions does not preclude th; use of cost and benefit 
Information in choosing whether and/or what kind of evaluation 
alternative to pursue. Even if evaluation benefits are more 
subjectively assessed, it is still important to establish how 
evaluation costs stack up against those benefits. As with costs, the 
resource plan provides the starting point for developing a checklist of 
benefits. ( Direct benefits being related to the kinds of I nformation 
outcomes provided by the evaluation and i ndirect benefits resulting 
from the evaluation process itself, as Illustrated on the previous 
page.) 

If alternative evaluation strategies are being considered, a thorough 
checklist of benefits for each alternative provides a richer context 
for weighing costs. In order to more closely compare different 
clusters of benefits, decision-makers can assign weights to each 
benefit as a rough way of measuring each benefit's intrinsic value to 
the evaluation user(s). Each evaluation alternative could then be 
measured In terms of a total benefit "score", as well as <otal 
evaluation cost. The more costly evaluation alternative may provide 
unique and highly valued benefits which significantly outstrip the 
potential benefits offered by less costly approaches. 

CONCLUSION 

The cost of an evaluation is an immediate an inescapable concern for 
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the 17PA planner and decision-maker. Is the evaluation doable, or will 
It cost too much? Often, however, this full scrutiny of evaluation 
costs is not equally applied to evaluation benefits. Although benefits 
may be less quantifiable and more subjectively felt than costs, they 
are nonetheless real, substantial and important in providing a fuller 
context for assess i ;g costs. 

In assessing costs and benefits, planners have to remain open to 
creative alternatives for carrying out a JTPA evaluation so tnat they 
do not feel locked in to a single, too costly plan. The next section 
explores some JTPA evaluation staffing and funding alternatives. 
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SECTION 4 
IMPLEMENTATION SSUES 



The preceding chapters trace the JTPA evaluation planning process, blending 
implementation issues into that process. How the evu'uation will be conducted, 
who will be involved, what information will be gathered- ihese are all implemen- 
tation issues that must be imbedded within the overaSi evaluation plan. The 
separation between planning issues and implementation issues is a somewhat 
artificial one, made here for clarity's sake. This final section pulls out three critical 
areas of evaluation implementation for closer examination: data collection issues, 
staffing concerns and funding options. 
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CHAPTER 8 
COLLECTING AND USING DATA 



What Is the Quality of the Data? 

Data Reliability 
Data Validity 
Data Comparability 

Are the Data Available? 

Will Different Data Sets Need to Be Merged? 

How Will Client Confidentiality Be Handled? 
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CHAPTER 8. 



COLLECTING AND USING DATA 



INTRODUCTIL % 

Whether data are derived from an HIS or other automated data base 
systems, access to accurate and valid data Is a key consideration In 
designing and Implementing any evaluation. Without adequate data, the 
most beautifully designed evalua Ion Is worthless. Evaluators should 
not wait until the final design and Implementation stage of an 
evaluation to plow through data gathering systems and be confronted 
with their Inadequacies. Rather, these systems should be explored and 
their Insufficiencies uncovered In the early evaluation planning stages. 

Many considerations besides analytical needs (e.g., political, 
technical, ethical) go Into the design of a data collection system. As 
a result, each system uniquely delimits what Information can 
Immediately feed Into evaluation. Given this diversity In HIS and 
other data systems across states and SDAs, the purpose In this chapter 
is to highlight those major data issues relevant to many JTPA 
programs. As with other evaluation concerns presented In this volume, 
data collection Issues have not only a technical face, but an 
organizational face as well. Each of the following Issues will be 
discussed In turn. : 

• What 1s the quality of the data? 

• Are the necessary data available? 

• Will different data sets need to be merged? 

• How will client conf Identlfilty be ha:-»'*led? 
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A more detailed discussion of MIS capabilities Important to evaluation 
activities will be presented in a later publication In this volume 
series . 

WHAT IS THE QUALITY OF THE DATA? 

For both monitoring and evaluation purposes, a primary concern Is the 
quality of the data. Quality rests principally on the reliability , 
validity , and comparability of the MIS and other data sets to be used 
In evaluation. 

Data Reliability 

Reliability has to do with the accuracy and consistency with which data 
have been collected. In the MIS, for example, there are severa" major 
sources of unreliable data: (1) the client himself or herself; (2) the 
staff who are recording Information on the client; (3) the data entry 
staff transferring that Information; (4) system classification schemes 
which do not clearly or consistently distinguish one data element 
category from another. In SDAs with highly decentralized intake and 
service delivery systems, the potential for data Inconsistencies and 
Inaccuracies Is multlpl led . In preparation for evaluation, planners 
can review data collection procedures and safeguards, recommending 
additional safeguards If necessary. 

Data Validity 

A related issue Is that of measurement validity : Do the data elements 
required in the evaluation truly measure what they are supposed to 
measure? For example, do simple "wages" truly represent "earnings" 
(the outcome JTPA legislation mandates for study)? If data on wages 
alone is used as an outcome measurement, other earnings , such as fringe 
benefits and tips, may be ignored. The analysis guides for net Impact, 
gross Impact, and process evaluation in Volumes 3, 4, and 5) deal 
further with validity Issues in the specific measurement context of 
each approach. 

Data Comparability 

Data collected within a single state or SDA may sufficiently meet 
standards of reliability and validity but still not be useful for JTPA 
program comparisons across states, SDAs, and even service providers. 
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In order to evaluate Implementation practices and outcomes within a 
broader state or regional context, the definitions of various HIS data 
elements need to be reasonably standardized. Achieving such 
standardization across different jurisdictions often proves to be a 
complicated task, especially in states which operate a more 
decentralized HIS system. 

Where JTPA services are decentralized through numerous separate 
contractors, the issue of data comparability extends all the way down 
to the service provider level. When the SDA (or proxy agency like the 
Employment Service) performs centralized intake and service assignment, 
it can perhaps exert more control over how participant information is 
categorized and codified in the HIS. But where these initial service 
functions are relinquished to independent contractors, standardization 
of information is more difficult to maintain. Rigorous categorizing 
and coding guidelines for contractors may not exist or, if they do 
exist, they may be hard to enforce at the contractor level. 

It is in the comparison of different program service strategies or 
treatments that HIS data comparability is often most questionable. 
National reporting requirements have led to states and SDAs collecting 
fairly standardized Information about JTPA enrollments, terminations 
and primary client characteristics. But because such reporting 
requirements are lacking for program variables (e.g., type of 
treatment, length of treatment) , treatment data can be much less 
uniform across states, SDAs and their individual service providers. 

When no standardized MIS definitions and coding guidelines exist, the 
definitions for various program treatments may be applied in 
non-standardized ways at every JTPA level-estate,. SDA, and individual 
service provider. Consider the category "pre-employment training." 
One service provider may lump into this category clients who are given 
a half-day course on job search techniques, along with clients taking a 
comprehensive three-week course. Another provider may categorize only 
the latter activity as "training", and regard the first activity 
primarily as "placement." This comparability problem will extend down 
to the individual service provider level in highly decentralized SDA 



EKLC 



98 

1ni 



where services are largely contracted out to numerous providers. 

Multiple treatment strategies may further complicate matters because of 
the added problem of defining which category gets credit for a 
resultant positive outcome. Crediting only the final treatment, as do 
some SDAs or their contractors, leads to a distortion of program 
outcomes: the success rate of the final treatment (often OJT) may 
appear greater than It actually merits because the costs per placement 
may be artificially deflated. Conversely, the costs per placement for 
all the more preparatory kinds of treatments (such as adult basic 
education, or skills training) may be over-inf lated, making these 
treatments appear less attractive. 

If HIS coding for these kinds of treatment variables are not 
standardized, a SDA wishing to Include such variables In an evaluation 
will have to establish clear guidelines for assigning treatment data to 
categories. Service providers may then be better able to translate 
their own coded data more appropriately to fit SDA ueflnltlons. 

ARE THE DATA AVAILABLE? 

In any state or local setting, the HIS, providing continuously 
generated Inft » matlon on a number of Important client and 
Implementation variables, will be a key factor In the evaluation. 
Besides data quality and comparability, a primary concern must be HIS 
sufficiency to meet the Important data requirements of evaluation. 
What demands. In fact, will evaluation place on the HIS? The different 
evaluation approaches presented In this series have different 
Information requirements wh1rh are detailed In each of the separate 
volumes of JTPA Evaluation at the Local Level . (In addition, a more 
specific discussion of the kinds of HIS capabilities which are 
desirable for JTPA evaluation will be provided In a forthcoming paper 
of this project. ) 

In ge'ieral, however, the various evaluation approaches will require the 
following basic categories of HIS data: 
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MIS DATA FOR EVALUATION 



1- Client characteristics 

• Age 

• Sex 

• Race 

• Etc. 

2. Service data 

• Type of treatment 

• Length of treatment 

3. Termination data 

4. Fo11ow-up data 

• Client data 

• Addltlon^il services 

5. Employer data 

• Employer l.D. Information 

• Employer services Information 

• Employer follow-up 

6. SDA/Communlty characteristics 

7. Financial data 



If the HIS lacks certain data elements useful to evaluation, how 
readily can the system be revised? It may be more cost-effective In 
the long-run to hammer out a thorough revision based on multiple 
evaluation uses, rather than slowly attack a system piecemeal. 

The cost of adding elements to the HIS Is an obvious constraint to 
modifying the system, in the more decentralized state settings where 
SDAs operate Independent software or mainframe systems, individual 
modifications may be especially costly because the states are likely to 
bear less responslbll Ity for local iv- run Information systems. But 
computer programming time Is not the only cost Issue Involved In 
acquiring new data for evaluation. SDAs need to be sensitive to the 
potential burdens that added reporting requirements will place on 
themselves and their service providers (designing new forms, training 
Intake personnel, etc.). Also, there Is a limit to how much research 
Information an SDA or service provider can collect without compromising 
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its social service mission. Therefore, part of initial evaluation 
planning must involve the integration of evaluation's MIS requirements 
into the SDA's overall information needs. 

In the more centralized state HIS systems, an SDA will have less 
latitude in independently modifying its HIS. Longer range planning for 
evaluation activities may entail bringing in the state and other SDAs 
to develop an HIS capability oriented towards local level evaluation 
needs. However, SDAs may have different information priorities from 
each other and from the state, complicating the task of enhancing the 
HIS to meet diverse evaluation needs. In some instances SDAs have 
collectively negotiated changes in proposed statewide evaluation to 
include gathering more information of direct concern to the SDAs. 

WILL DIFFERENT DATA SETS NEED TO BE MERGED? 

While HIS information will often be aic the core of many JTPA 
evaluations, additional information may also be critical. For example, 
merging HIS client data with other kinds of client data on post-JTPA 
earnings, employment and welfare dependency permits a more 
sophisticated analysis of program outcomes and impacts. 

Frequently this additional kind of data is contained in data base 
systems completely separate and incompatible with the JTPA HIS. The 
evaluation plan should anticipate the technical difficulties to be 
overcome in bringing various data systems together for a unitary 
analysis . 

Technical difficulties in merging data are not confined to the managing 
of different computer systems and programs; the data itself may present 
stumbling blocks. For example, in some states the category 
"disadvantaged" is not flagged in the Employment Service registrants 
data base used in the net impact evaluation to construct a comparison 
group. Lacking this category, it will be more difficult to match and 
compare JTPA participants with similar groups of non-participants. 
(See Volume 5, Chapter 4 for an in-depth treatment of this concern.) 
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The task of merging MIS with other kinds of data can Involve 
organizational considerations, as weil. The data may be under another 
agency's authority, and obtaining that data may pose additional 
challenges. Commonly, data requests across agency boundaries are 
viewed as an Imposition, requiring extra staff time or other 
resources. If the lines of communication between agencies are poor, 
the data collection effort may suffer. 

Moreover, the agency may be under state rather than local jurisdiction 
(e.g., welfare). There may. In this case, be less organizational 
precedence or support for Interaction and cooperation with the SDA. If 
state JTPA administrators are also Interested In the non-JTPA data for 
their own purposes. It may be easier for the SDA to work through the 
state JTPA auspices to obtain the data. 

Such realities underscore the need for strategic organizational 
planning as part of the overall evaluation planning effort. 
Representatives of affected agencies should be brought Into the 
planning process early to ensure greater cooperation. Any Interagency 
understandings about data sharing and computer use should be put In 
writing as further Insurance against future frustrations and 
m1sund»jrstand1ng. 

HOW WILL CLISNT CONFIDENTIALITY BE HANDLED? 

Although state agencies and SDAs may routinely share JTPA client 
information, client confidentiality is not an issue as long as that 
Information is presented in the aggregate without individual 
identifiers, such as client name or social security number. However, 
both the net and gross Impact evaluations Involve the merging of MIS 
data with other sources of data for which client identifiers are 
required to accomplish the match of information. 

To implement an evaluation, two or more separate agencies may have to 
share JTPA data flagged with client identifies. Each agency may have 
Its own internal standards regarding client data access and use. For 
example, one agency may strictly limit information containing client 
identifiers to a small number of special users, while others may allow 



ERIC 



102 



wide access to such information. Some agencies may permit client data 
to be used for compliance investigation and others may not. In such 
cases, interagency discussion and agreement about client 
confidentiality must be part of the evaluation planning effort. 

Assurances about client confidentiality may be especially important to 
SDA3 and service providers. Inability to ensure client confidentiality 
may impair the client-service-provider relationship and subsequently 
impact treatment success. Breaches in client confidentiality may also 
discourage eligibles from participating in JTPA. For these reasons, 
policies regarding the use of evaluation data need to be established in 
advance. 

CONCLUSION 

Evaluation findings are only as good as the information foundation they 
rest on. If data are incomplete, unreliable or inaccessible, 
evaluation resources may be unnecessarily wasted or the evaluation's 
utility substantially compromised. As a preventative measure, the 
evaluation plan should incorporate a review of relevant data collection 
procedures and data access systems. Such a review addresses not only 
methodological concerns (data accuracy, reliability, validity, 
comparability) and technical concerns (data availabil ity, computer 
capabilities), but also organizational concerns (data sharing, client 
confidentiality). In meeting these concerns, an SDA is not only better 
prepared to implement evaluation, but also enjoys certain long-term 
benefits in terms of increased data-collection efficiency and accuracy 
affecting other oversight and research activities. 
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CHAPTER 9 
FUNDING A JTPA EVALUATION 

What Are JTPA-Rslated Sources of Funding? 

What are Other Sources of Funding? 
What Funding Strategy Shouid be Pursued? 
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CHAPTER 9. 
FUNDING A JTPA EVALUATION 



INTRODUCTION 

While JTPA legislation supports various evaluation activities, no 
specific funds are allocated to this purpose. As long as 
administrative funds remain so limited, finding financial support for 
JTPA evaluation will be a fundamental concern for most states and SDAs. 

The purpose of this chapter Is to encourage local JTPA planners and 
decision-makers to think broadly and creatively about funding 
possibilities for JTPA evaluation. The JTPA's orientation toward 
public-private collaboration In addressing employment and training 
needs sets the stage for exploring new funding partnerships In the 
evaluation of JTPA programs. Before examining these new partnership 
possibilities, we briefly outline various sources of support for 
evaluation within JTPA. 

WHAT ARE JTPA-RELATED SOURCES OF FUNDING? 

Under current JTPA formula-funding levels, most SDA's Internal 
resources for doing evaluation will be limited. Beyond the 15X 
administrative monies allowable under the Act, SDAs may want to explore 
pooling administrative resources from other pots of JTPA money to carry 
out evaluation activities. 

JTPA Special Set-Asides 

In looking for sources of evaluation funding, an obvious place to start 
Is with the JTPA state set-asides designated for special 
admlnlstratalve and other activities. In many states, SDAs are moving 
towards collective negotiation and cooperative decision-making as to 
how these state funds might be allocated. While use of these funds for 
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evaluation may be restricted In various ways, a portion of the six 
percent, three percent and eight percent pots of money might arguably 
be applied to pertinent evaluation efforts- The evaluation-related 
possibilities for each of these set-asides are outlined here. 



• Six Percent Set^Aslde : 

Much debate has already ensued around the appropriate use of six 
percent monies for technical assistance to SDAs. The debate 
centers around what, precisely, "technical assistance" (a term 
net specifically defined In the legislation) can encompass. Is 
evaluation an acceptable form of technical assistance? In the 
past, the Department of Labor (DOL) has questioned the use of 
six percent monies for state evaluation activities because the 
legislation directs states to offer technical assistance to 
those Individual SDAs who are falling to meet performance 
standards . As of this writing, however, DOL has not taken a 
firm position, allowing states discretion on this Issue. 

Using this discretion, some states have Interpreted the six 
percent more broadly to allow for evaluation. The argument here 
Is three-fold: First, states cannot adequately develop 
technical assistance packages to SDAs without first having a 
means to assess what Is or Is not effective about JTPA both 
generally and specifically at the SDA program level. Evaluation 
activities provide the necessary Information base for 
Implementing useful technical assistance. 

Second, the legislation Intended performance standards measures 
to be selective Indicators of how well JTPA programs are meeting 
certain goals, not comprehensive measures of JTPA goal 
achievement. Therefore, the purpose of technical assistance 
activities such as evaluation need not be directly and narrowly 
tied to Improving performance measures , but rather should be 
related to improving the program's effectiveness In meeting Its 
Intended goals. 

Finally, evaluation helps spot program difficulties before they 
are reflected in performance measures, allowing for more timely 
correction of problems. Evaluation therefore, may be viewed as 
"preventative" technical assistance to SDAs who might otherwise 
fail to meet standards. 

Pending a restrictive federal definition of technical assistance 
and the circumstances under which such assistance can be 
provided, states and SDAs might explore the use of six percent 
set aside monies for supporting evaluation activities as a form 
of technical assistance. 

• Eight Percent Set>>As1de : 

JTPA requires that eight percent of state funds be set aside for 
state education and coordination activities. While in many 
Instances, states are retaining complete control over these 
funds, in other cases, states are using SDAs as conduit for the 
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funds going to special contractors. Since up to one-fifth of 
these funds may go towards enhancing JTPA coordination, local 
JTPA administrators may be able to make a case for tapping a 
portion of these funds to do evaluation which focuses on program 
coordination aspects of concern to both SDAs and the state. 

• Three Percent Set-Aside : 

Three percent of state administrative funds are set aside for 
special programs and services to disadvantaged older workers. 
This Is the smallest pot of state set-aside monies, and where 
program funds are funneled through the SDAs the 15 percent 
administrative restriction applies. Nonetheless, In some 
Instances, It may be feasible to use a percentage of these funds 
to evaluate special JTPA activities for older workers. 

Title III Funds 

In many states, the SDA role In Title III has been fairly limited. But 
since some SDAs manage Title III programs, and SDAs In general are 
Increasingly Interested In participating more fully In such programs, 
some thoughts on Title III evaluation funding are Included here. 

Both Title III formula funding and discretionary funding allow 
significant administrative flexibility to support evaluation 
activities. In order to receive formula-allocated funds, states must 
match federal funds with their own program funds or In-kind support. 
In states with greater unemployment, the match regulrement Is 
proportionately reduced. While 70 percent of funds must go to direct 
service, this limitation applies only to federal funds and only up to 
50 percent of aJM program funds combined. These provisions give states 
and SDAs considerable latitude to Incorporate evaluation Into Title III 
activities. Evaluation costs may be counted as state match money; more 
1 Iberal limits on administrative costs In general make support for 
evaluation more feasible. 

Title III discretionary funds which the Secretary of Labor manages are 
to support special state training programs In areas of high 
unemployment, plant closures and mass layoffs. Since no state match 
money 1s required and no specific legislative limitations are placed on 
these funds, states and SDAs have a special opportunity to Integrate 
evaluation Into training programs sponsored by these funds . Because 
state and local program activities geared towards dislocated workers 
are a relatively new phenomenon, the rationale for building evaluation 
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tooruinaxion aspecis or concern to Dotn bUAs and the state. 

• Three Percent Set>As1de : 

Three percent of state administrative funds are set aside for 
special programs and services to disadvantaged older workers. 
This Is the smallest pot of state set-aside monies, and where 
program funds are funneled through the SDAs the 15 percent 
administrative restriction applies. Nonetheless, In some 
Instances, It may be feasible to use a percentage of these funds 
to evaluate special JTPA activities for older workers. 

Title III Funds 

In many states, the SDA role In Title III has been fairly limited. But 
since some SDAs manage Title III programs, and SDAs In general are 
Increasingly Interested In participating more fully In such programs, 
some thoughts on Title III evaluation funding are Included here. 

Both Title III formula funding and discretionary funding allow 
significant administrative flexibility to support evaluation 
activities. In order to receive formula-allocated funds, states must 
match federal funds with their own program funds or In-kind support. 
In states with greater unemployment^ the match regulrement Is 
proportionately reduced. While 70 percent of funds must go to direct 
service, this limitation applies only to federal funds and only up to 
50 percent of all program funds combined. These provisions give states 
and SDAs considerable latitude to Incorporate evaluation Into Title III 
activities. Evaluation costs may be counted as state match money; more 
liberal limits on administrative costs In general make support for 
evaluation more feasible. 

Title III discretionary funds which the Secretary of Labor manages are 
to support special state training programs In areas of high 
unemployment, plant closures and mass layoffs. Since no state match 
money Is required and no specific legislative limitations are placed on 
these funds, states and SDAs have a special opportunity to Integrate 
evaluation Into training programs sponsored by these funds. Because 
state and local program activities geared towards dislocated workers 
are a relatively new phenomenon, the rationale for building evaluation 
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Ultimately, however, casting a broader net Into funding realms beyond 
the familiar can pay off in many ways. Even if adventuresome searchers 
are not directly rewarded with the cash support they seek, the effort 
may still provo valuable in terms of non>monetary contributions, 
increased contacts and interactions within the business, academic and 
professional communities, increased program visibility and credibility, 
and enlarged possibilities for future funding. The remainder of the 
chapter outlines some of these alternative funding possibilities. 

Universities and 4--year colleges 

Academic institutions can often offer unique evaluation resources at 
reduced costs. First, a major academic resource is faculty who may 
have the specialized research expertise needed, and are often available 
at a reduced cost compared to private consultants . Through their 
institutional ties , faculty are sometimes better able to leverage 
related research resources (such as research materials, computer 
expertise, other faculty and students). If the faculty consultant time 
commitment is below a certain amount, academic institutions will often 
reduce or waive their indirect cdsts. 



Students are another potential source of support for evaluation. 
Frequently, graduate students are willing to devote research time to an 
outside evaluation project in order to gain practical work experience 
(encouraged or required by many professional graduate schools) or to 
develop material for a thesis or doctoral project. Sometimes students 
(as well as faculty) can partially or fully support their evaluation 
research activities through research assistantships , post- doctoral 
fellowships or individual research grants. While very limited, federal 
work study funds do exist at the graduate level, allowing employers to 
pay only a portion of the wage costs of a work study student. An added 
plus is that students bring with them the advice, interest and support 
of supervising faculty who can act as an additional quality control on 
the student's work, and who themselves may be willing to play an active 
role in the evaluation effort, contributing specialized expertise. 

State-supported educational institutions (including community colleges) 
are also part of the state agency network. Their public-sector status 
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provides an opportunity and rationale for developing closer ties that 
can be mutually beneficial to both parties. In terms of hiring a JTPA 
evaluation consultant, contracting with state-supported colleges or 
universities may be simpler, less formal , and Involve lower Indirect 
costs than would other contracting arrangements. 

For a variety of reasons, academic departments are frequently 
Interested In setting up formal ties with agencies sponsoring research 
projects. Such ties might take the form of special Internships for 
qualified students or reduced-fee faculty consulting. In some cases, 
graduate departments or professional schools may partially or fully 
fund studies of evaluation Issues of special relevance to their faculty 
and students. One local JTPA evaluation, for Instance, was largely 
sponsored by a nearby university's graduate business school. Faculty 
and SDA staff planned the evaluation; students collected and analyzed 
data under faculty supervision. When collaboration with a university 
Is more formalized, faculty are more likely to play an active role In 
screening and supervising students. 

Special Organizations 

A number of non-profit business, labor, professional, social service 
and public Interest organizations have a special Interest In ev<iluating 
and thereby Improving employment and training programs. A JTPA 
evaluation may be able to capitalize on this Interest In a number of 
ways . For example, members of such groups might act as formal or 
Informal advisors to the evaluation planning process. Members might be 
w1 1 1 1ng to offer reduced fee services or provide certain resources In 
exchange for public recognition of their contributions. The National 
Alliance of Business (NAB), for example, has contributed to local JTPA 
evaluation activities. Other organizations might also be willing to 
lend various forms of support. 

Private Foundations. Charitable Organizations and Trusts 
Private foundation support used to be almost entirely the preserve of 
educational Institutions and non-profit organizations. Increasingly, 
however, public agencies have broadened their funding si:rateg1es to 
Include soliciting foundations for support. Nor 1s foundation support 
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limited to direct services; many foundations are concerned with 
developing Innovative approaches to service delivery and are willing to 
fund applied research activities (such as evaluation) In a number of 
service areas, Including employment and training. 

Host major metropolitan libraries carry standard directories (refer to 
the reference section for examples of these directories) profiling the 
larger national and regional foundations and their giving patterns- 
Regional directories of state and local funders are also usually 
available. Such directories provide Initial Information needed to 
Identify those funders whc are most likely to bf. Interested In social 
program evaluation activities and )n employment and training Issues. 

The major directories Include fairly detailed and historical profiles 
on foundation activities (previous funding patterns, kinds of costs 
covered, special requirements, current recipients of support), which 
help the researcher quickly narrow the search effort. Financial 
"reports of foundations, charities, and trusts within a state also give 
a good sense of who and what these organizations fund, their funding 
philosophy and agenda. (These reports are generally available through 
the state attorney general's office or the state agency which oversees 
the financial reporting of charitable organizations.) 

These funders may be more attracted to programs which are Innovative or 
can serve as demonstration models for other programs. Evaluation of 
programs geared to special populations (e.g., youth, ex-offenders, 
welfare recipients, older workers) may also resonate with certain 
funders who otherwise would not be Involved with JTPA evaluation 
activities. 

Size and location of foundations are often Important considerations. 
Smaller and more local foundations may be more unpredictable In their 
outlook, but they will also be more geared to local actors and 
Interests . They may support an especially appealing local project 
outside their usual framework. 

In contrast, the larger national foundations are more bureaucratic, 
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engage 1n a very formalized selection of issues to be funded, have more 
specifically defined application procedures and fixed funding 
parameters, and apply more rigid criteria ir making funding decisions. 
Larger foundations also tend to have lengthy timeframes for review and 
final decision-making. The trade-off is that major foundation support, 
while more competitively sought, more difficult and time-consuming to 
achieve, offers larger pots of money, greater prestige and increased 
likelihood of supplemental funding in the future. Therefore, while an 
SDA's best chances for funding may be at the local level, the fund 
seeker should not automatically preclude national and state: funders. 

Private Business Sector 

JTPA envisions a close working relation between government and the 
private sector to better connect those who are being trained with those 
who can offer jobs. In the interest of learning more about and 
improving current JTPA operations, the public-private partnership might 
arguably be extended to include joint support for evaluation activities. 

Large companies utilizing JTPA services such as OJT may be particularly 
receptive to requests for assistance in evaluating and improving those 
services. (More support may be available 1f the company also views its 
participation in terms of public relations returns.) While local 
service agencies may be unaccustomed to approaching the private sector 
directly for help, a mechanism for making such contacts is built into 
JTPA through the PICs. The project which sponsored this set of 
evaluation guidebooks is a prime example of how private businesses may 
join with the public sector in supporting evaluation activities. 

In addition to approaching business contacts through JTPA channels, 
other sources of information on private sector companies are available 
to help in the fund search. State employment agencies, economic 
development organizations and private research companies often publish 
Information on the largest employers in the state. Also, major 
university and public libraries usually carry reference guides on 
corporations in each state, which describe their giving programs. (See 
the final section for specific references.) 
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Local companies can be contacted directly for Information about their 
funding Interests and requirements. Usually, the funding proposals do 
not need to be as long or as complex as with other funders and the 
decision time Is much shorter. 

With major national corporations, the scenario can be quite different. 
They often have special (usually non-local) corporate giving units that 
handle all funding requests, often requiring somewhat more 
sophisticated and detailed proposals. While these special units may 
make the final selections, local corporate branches may also wish to be 
Involved In the review process and may have Influence over the ultimate 
funding decision of corporate headquarters. 

WHAT FUNDING STRATEGY SHOULD BE PURSUED? 

Funding sources of all kinds have reduced their giving programs over 
the past few years. Creative, Imaginative and well thought-out funding 
strategies have always made a difference, but now they are Imperative. 
In the present period of scarcity and shifting social welfare values, 
funding social services Is a genuine challenge. It Is also 
Increasingly difficult to locate funders with a special Interest In the 
assessment of employment and training efforts. Therefore, the fund 
searcher must build maximum efficiency Into the fund search effort. 
Following are some strategies for developing JTPA funding proposals and 
Increasing the likelihood of their success. 

1. Identify potential funders of policy research, 
particularly ongoing program evaluation or the 
evaluation of pilot and demonstration programs in 
the human services. 

The economics and business sections of most public libraries have 

excellent books on foundations and corporations, produced by major fund 

search organizations and publishers of business/Industrial 

directories. Repositories for government documents In colleges, 

universities, and state libraries have Information on government 

funders. The Grants and Contracts Weekly and The Business and Commerce 

Dally are the most current sources of information on government funding 

priorities. Automated searches, now avallble at a low cost to 

government agencies, provide quick sources of Information on a range of 
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private and public sector funders. A large pool of "possible" funders 
can therefore be compiled from the rich resources now readily available, 

2. Develop fund search criteria, which help you narrow 
fund search efforts to the most likely funders. 

One way to economise In the search effort Is to narrow these possible 

funders to the probable ones, and the probable to the most likely, 

prior to any extensive fund search effort. To conserve energy, It Is 

helpful to first become as knowledgable as possible about 1) the 

characteristics of what you want funded, and 2) the characteristics of 

the potential funders you have sifted out In your first review, and 

then apply these criteria as a guide In matching your project's 

features with the most Hkely subgroup of funders for that particular 

project. Some useful criteria are listed below. 



CRITERIA FOR SELECTING FUNDERS 



Project Characteristics 

• The purpose of the evaluation 

• The primary Issues to be Investigated 

• The kinds of groups and/or organizations to be studied 

• The nature of the sponsor of the evaluation activities 

• The context In which the evaluation Is to be carried out 

• The kind of support already acquired for the evaluation 

Characteristics of Potential Funder 

• The funder's source of funds 

• The size of the funder 's total giving program, and the 
average amount contributed to any given recipient 

« The funder's relationship to the source of Its funds 

• The public/private status of the funder 

• The funder*s size and level of bureaucratization 

• The funder*s historical and current funding pattern with 
respect to: 

^ The kinds of Issues emphasized 

^ the types of recipients funded 

the degree of formality, sophistication, and complexity 
of application, review and selection processes 

• The geographic diversity of the recipients 
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3. Develop a General Fund Search strategy. 

Work plans and tlmefrjimes are generic to government. They are just 
as necessary for fund search as for the development of the projects 
for which funding Is sought. Although In the final analysis, such 
plans and timetables must be tailored to different funders, It Is 
helpful to begin with an overall strategy which Is modifiable. Such 
a stategy has at least the following elements: 

• The preparation and/or acquisition of basic fund search materials 

• The preparation of a general description of the project to be 
funded 

• The securing of general letters of support for the project, from 
Individuals whose endorsement will likely Increase the 
credibility of the funding request. 

• The preparation of a general cover lettet to accompany these 
materials 

• The development of a chronological work plan and timeframe for 
obtaining funds, based on considerations of staff resources, 
time pressures, the need to acquire funds from more than one 
source, the ability to maintain organizational support from the 
project sponsor over time, the realities of governmental and 
nongovernmental funding cycles, and other organizational and 
political considerations. 

4. Fine-tune the materials to each of a small set of 
top priority funders within the "likely" group. 

The most Important aspect of tailoring a funding request Is to achieve 

an honest mesh between the characteristics of the proposed project and 

the current priorities of the funder (and to a lesser extent the 

funder's historical giving pattern). The goal Is to construct an 

Individualized funding rationale for each potential funder to be 

approached. 



5. Decide which is more appropriate: A single-f under 
or multi-funder approach. 

If the latter, each funder should be Informed In the cover letter what 

others are being simultaneously approached. Learning this from other 

funders Is often the kiss of death for a funding application. A 

staggered approach to multiple funders may In some cases be the best 

method. Securing one major funder may tend to leverage funds from the 

others. Corporate givers may be resistant to being the only private 
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funder; a mult1-funder strategy should consider incoroporating more 
than one private funding proposal. 

6. Update critical information on the funder(c) 
selected for the first phase of the fund search. 

You may need to solicit fresh information on funders through personal 

contact with an individual within the funding organization who is in a 

position to give you the information you need relative to the kind of 

project being proposed. It is wise, however, not to identify the 

project or its sponsor at this point, since this can affect funding 

decisions prematurely. Rather, this should be a general 

information-gathering call, confined to questions such as: 

• What are the funding application guidelines and procedures? 

• Where and to whom does one submit an application for funds? 
(Application materials should be requested, since many funders 
require a high level of conformance with their formal 
procedures.) 

• What are the current funding priorities? (Some corporations and 
smaller foundations wi 11 not tell you. The larger foundations 
have elaborate booklets outlining and justifying their current 
areas of interest.) 

• How flexible is the application process? 

• What additional factors may feed into the selection process? 

• Are public sector programs likely to be considered seriously for 
awards? 

• Who is the best contact person for following up on the status of 
an application? 

• What other kinds of things will the funder look for in an 
application? 

In studying these specific characteristics of funders, searchers will 
then be better able to further narrow the fund search to a few select 
and optimal choices. 

7. Identify special internal resources and 
capabilities. 

As part of a well-crafted proposal, a fund-seeker will want to 
emphasize those specific organizational resources and capabilities 
which will positively affect the evaluation process. Funders will be 
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looking for special characteristics that set the fund-seeking 
organization apart, characteristics that suggest the organization will 
be able to carry out the proposal In a successful, effective manner. 

For example, many funders are Impressed with proposals that appear to 
marshal effective community support or Involvement or that have already 
obtained contingency funding from other sources. Also, some funders 
may favorably view projects for which consultants or 1n~house staff 
with requisite training and experience for the project have already 
been Identified by fund-seekers. 

8. Develop cooperative relations with organizations 
which can act as funding brokers. 

Private sector funders at any level are likely to be resistant to 

funnel Ing support directly to public sector agencies or local 

governmental units. JTPA fund searchers will therefore, want to 

explore the use of "funding brokers" for their proposals. Such brokers 

might Include relevant university departments, research Institutes, or 

an appropriate non-profit organization which agrees to pass through the 

funds to the SDA. In exchange, the broker may expect some level of 

participation In the project or may charge for Indirect costs In acting 

as a funding conduit. 
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9. Solicit powerful advocates who can call or write to 
the funder on behalf of the project at an 
appropriate point in the review process. 

This Is a sensitive Issue which must be carefully handled and timed. 

Too much and too little advocacy can be a problem. 

10. Submit funding applications to the preferred 
funders, followed by a call to contacts within the 
funding organization to check on their receipt of 
the application and to clarify the review and 
selection process. 

Applications can take the form of finely crafted cover letters 

accompanying long proposals conforming to a myriad of strict 

guidelines, or they can Involve brief cover letters oriented to the 

funder's primary funding purposes accompanying a short concise concept 

paper on the project and why It Is In the Interest of the funder to 

support It. Whatever the format, the rationale for a particular funder 

to support a project must be clearly presented. 
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11. Wait patiently for an acknowledgement that your 
application has been received and for most of the 
review process to have taken place , and then 
implement an advocacy effort. 

Host large foundations politely notify the applicant and keep the fund- 
searcher Informed about the process. Most corporations do not. 
Smaller foundations and employers are often very amenable to calls from 
applicants. 



The source of advocacy Is Important. Pressure from elected officials 
may work well with government agt^ncles but not necessarily with private 
foundations or corporations. Local foundations are affected by 
advocacy from the client groups Involved, or from client advocacy 
groups In the community. 

12. If the first wave of fund search activities fail, 
select another set from the "likely" pool and begin 
again. 

You will want to seek Information on why the first choices turned you 
down. This may help you revise your concept papers and proposals, as 
you tailor them for new funders. In gearing up for another round, you 
may want to consider a different kind of f under- - smaller, or closer to 
the project, or go the other direction. You may prefer a 
private/public partnership strategy this time. If you tried for a 
single f under the first t\me. Skill , Imagination, flexibility, 
patience, and confidence In you project are essential In modifying your 
general funding strategy to accomodate for the normal series of wins 
and losses In fund search. 



CONCLUSION 

For many SDAs Interested In evaluating JTPA, funding will be an 
Important preliminary hurdle to negotiate. While new evaluation 
responsibilities have fallen on SDAs, traditional government funding 
sources under JTPA are far more limited than in the days of CETA. As a 
result, funding strategies may have to rest more on combining financial 
and in>kind support from several funding sources. Various JTPA-related 
administrative pots of money are obvious sources for partial funding of 
limited JTPA evaluation activities. For some SDAs , joint funding 
arrangements within or across JTPA- involved agencies may prove the most 
feasible way to sustain an ongoing evaluation capability. 
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Non«trad1t1onal funding sources should not be overlooked. Public 
sector administrators, generally unaccustomed to venturing beyond 
government funding options, will clearly have to move towards engaging 
support outside as well ai Inside the public sector. Universities, 
professional and community organisations, business and labor groups, 
private? foundations and corporations may represent Important untapped 
resources for carrying out JTPA evaluation. 
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CHAPTER 10 
STAFFING A JTPA EVALUATION 



Will Evaluation Require Special Staffing? 

Who Sliould Staff an Evaluation? 

The In-House Spproach 
The Outside Consultant Strategy 
Compromise Strategies 

Consultants: Wliat Are the Options? 

Finding and Selecting a Consultant 
Contract Concerns 
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CHAPTER 10. 



STAFFING A JTPA EVALUATION 



INTRODUCTION 

Because each SOA will have Its own evaluation Interests and needs, 
every evaluation effort will be somewhat unique; no single staffing 
pattern suffices for all. In some settings, an In-house team of 
evaluation specialists Is most feasible; In other contexts, an outside 
consultant may make more sense. Each approach has potential advantages 
and disadvantages which will be outlined later In this chapter. An 
important consideration Is whether available in-house staff have the 
technical skills to accomplish the kinds of evaluation tasks that are 
required. In addressing this consideration, we look first at some of 
the specialized staffing needs an evaluation might entail. 

WILL EVALUATION REQUIRE SPECIAL STAFFING? 

Comprehensive evaluations will likely require evaluation specialists In 
areas such as research design and statistical analysis; more 
scaled-down efforts might manage with fewer expert resources acting In 
a more limited consultant fashion. Whatever the scale, most 
evaluations will require some special staffing. The charts which 
follow present a rough notion of the sorts of special staffing needs an 
evaluation might engender: 
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CORE EVALUATION STAFF 



Type of 
Specialist 



Examples of 
Specialist Activities 



Program Evaliiator 
(specializing 1n employment 
and training programs) 



Develops and Implements a feasible 
overall evaluation approach (the 
questions to be Investigated) and 
methodology to meet the Information 
needs of a state or SDA. 



Coordinator of Evaluation 
Activities 



Coordinates activities In support of 
evaluation. Assesses the supports and 
constraints for conducting evaluation; 
develops strategies for Increasing the 
utility and utilization of evaluation. 
Coordinates activities across agency 
and division boundaries. Plans and/or 
coordinates resource utilization, 
staffing, and other Implementation 
components of the evaluation. 



HIS Programmer/ 
Analyst 



Develops programs needed for merging 
categorical data from different sources 
Creates customized data sets for 
analysis purposes and does data 
analysis under the supervision of the 
program evaluator. 



Surveyor, Interviewer or 
other Data Collectors 



Carries out the actual collection of 
Information required by the evaluation 
research approach. 
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ADDITIONAL EVALUATION SPECIALISTS 



Type of 
Specialist 



Evaluation Researcher 
(specializing In 
evaluablllty assessment) 



Examples of 
of Specialist Activities 



Determines the feasibility of carrying 
out different kinds of program eval- 
uations, given a state or SDA's 
evaluation needs. 



Research Design Specialist 



Sampling Specialist 



Advises a program evaluator on the most 
appropriate and efficient strategies 
for data collection and analysis. 

Advises program evaluator on sampling 
strategies to ensure maximum validity 
and reliability of Information 
collected. 



Survey Researcher 



Advises on the construction of 
Interviews and questionnaires. Assists 
In Implementation of phone, mall, or In- 
person surveys of JTPA participants, 
employers and others. Trains and super- 
vises Interviewers. 



Applied Social Statistician 



Advises on appropriate and efficient 
methods for statistical analysis of data 
In order to obtain valid Information. 



Public Information Staffer 



Assists In promotion of evaluation 
effort, developing Informational 
materials and/or funding solicitations. 
Assists In packaging and dissemination 
of final reports. 
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At first glance, the above 11st of specialized staffing needs may seem 
formidable. However, the list Is offered not to discourage, but to 
realistically present some of the distinct resources evaluation will 
have to draw on In order to provide truly useful Information about 
JTPA. The experts listed In the second chart (Additional Evaluation 
Specialists) are necessary only If the evaluation questions to be 
answered present particular research challenges where the core staff 
must turn for special advice. Moreover, a small core research staff 
can encompass a number of these skills so that staffing costs need not 
be prohibitive. One state for example, accomplishes much of Its 
ongoing JTPA evaluation work with one research director and two 
assistants . 

WHO SHOULD STAFF AN EVALUATION? 

Two major staffing configurations for carrying out evaluation are 
possible: In-house staffing and outside consultant staffing. Each has 
Its decided pluses and minuses, which will be more or less pronouced 
depending on the particular evaluation context. The following 
discussion touches on the potential advantages and disadvantages of 
each staffing approach and offers some compromise strategies combining 
both. We begin with an examination of the In-house staffing approach. 

The In-House Approach 

Some states and SDAs are meeting the JTPA evaluation challenge through 
creative In-house approaches. While many SDAs or their CETA 
predecessors never have themselves conducted comprehensive evaluation 
of their employment and training programs, they often have access to 
untapped resources sufficient for such an undertaking. In larger SDAs, 
although requisite staff may be scattered throughout the JTPA or local 
government systems, these resources may be drawn together as a special 
evaluation team or loosely coordinated as an In-house consultant panel. 

Certainly, cost Is one of the most compelling arguments for seeking 
in-house expertise. However, In certain settings, such an approach may 
Involve many hidden costs which need to be entered into the overall 
calculation in deciding which staffing strategy to pursue. 
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First, 1t may take significant time and effort to locate and engage 
special evaluation staff. Division or agency heads are likely to be 
skeptical and resistant to loaning personnel (underscoring what has 
been said earlier about the Importance of building broad organizational 
support for evaluation). Also, pooling In-house staff resources may 
require extra management staff to bridge the commu.jicatlon and 
coordination gaps that Inevitably will arise. And finally, there may 
be some Inefficiencies associated with less experienced and less 
specialized staff attempting to progress along a learning curve while 
evaluating JTPA. 

Cutting corners on evaluation specialists may ultimately cost the 
organization far more than would have originally been spent on 
consultant fees. Where In-house evaluation staff lack requisite 
technical expertise, the great risk Is that the Information obtained 
will lack sufficient reliability or validity; the findings will be of 
diminished value. A less obvious cost of using In-house evaluators may 
be lower credibility of the evaluation results. 

However, the In-house approach to evaluation also carries some less 
obvious, but potenllaliy Important benefits, which Include: 

• Evaluation staff's familiarity with the organization setting, 
data collection systems, staff capabilities, time schedules, 
program procedures, etc. 

• Fewer entry problems for evaluation staff, more rapport with 
program staff; greater receptivity to programmatic needs of 
staff. 

• Cost savings potential through closer monitoring and control cf 
the work In progress. 

• Opportunities to foster Inter- and Intra-agency communication. 

• Capacity-building for further evaluation efforts. 

• Flexibility In reassigning evaluation staff to evolving tasks. 

In-house staff may clso provide continuity to the evaluation process. 
Staff are present at the beginning, so that evaluation needs are 
accomodated in program design and evaluation; staff are also present 
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after the evaluation, to facilitate and encourage the programmatic 
changes identified as useful. 



Building an In-House Evaluation Capability 

In building a JTPA evaluation capability, SDAs have a number of 
options . Given the wide range of evaluation needs , in-house 
capabilities, and organizational constraints in each SDA, no one option 
can claim clear superiority. The staffing approach that is effective 
in one setting, may be ineffective in another. Of particular concern 
in assessing the appropriateness of a staffing straiiegy are the six 
criteria mentioned in the Chapter 4 disctission of where to locate an 
evaluation unit. Again, those criteria are: 

• Position within the authority structure 

• Separation from compliance functions 

• Neutrality 

• Trust 

• Coordination capabilities 

• General competency 

When applied to different staffing approaches, these criteria suggest 
pluses and minuses and distinct tradeoffs between those approaches. 
Each SDA will have to judge for itself how it may best develop its 
evaluation capabilities, given the organizational framework within 
which its JTPA programs operate. For a specific checklist of concerns 
about who should do evaluation, see the following page. 

The Outside Consultant Strategy 

Within the evaluation community the debate over whether to use in~house 
resources has been ongoing. Obviously, in circumstances where access 
to in~house expertise is limited, turning to outside evaluation 
specialists is the only option. 

However, critics of the in-house approach argue that even if in-house 
resources are available, some important potential benefits offered by 
outside consultants should not be overlooked. These potential benefits 
include: 
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WHO SHOULD DO THE EVALUATION? 



AUTHORITY FACTORS 

1. Will evaluation staff be removed from the 
organizational hierarchy of programs being evaluated? 

2. Will evaluation staff report directly to key 
decision-makers? 

3. Will evaluation activities be separate from 
compliance activities? 

4. Will evaluation staff h^ve sufficient status to 
obtain necessary cooperation with program staff? 

INDEPENDENCE AND NEUTRALITY FACTORS 

1. Will evaluators' organizational status permit their 
Independent Judgment and action where appropriate? 

2. Will those being evaluated receive evaluators as 
independent and neutral? 

3. Will funders or other decision-makers view 
evaluators as independent and credible researchers? 

TRUST FACTORS 

1. Will evaluation staff have requisite interpersonal 
skills? 

2. Will evaluation staff have good rapport with program 
staff and ready access to information? 

3. Will decision-makers be likely to trust the 
evaluation findings? 

COMPETENCY AND COORDINATION FACTORS 

1. Will evaluation staff have requisite skills/ 
expertise? 

2. Will evaluation staff include those with specific 
experience in evaluating employment and training 
programs? 

3. Will evaluation staff be familiar with the JTPA 
system? 

4. Will evaluation staff include those with good 
organizational, planning and management skills? 

5. Will evaluation staff be able to effectively use 
and develop communication and coordination channels 
among JTPA actors? 



YES NO NOT SURE 
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• Greater credibility with evaluation users, particularly funders. 

• Separation from the organization which allows for greater 
objectivity and fairness (actual or perceived). 

• Here acceptance from program staff who feel less threatened. 

• Greater assurances of a quality product produced by an experienced 
specialist. 

• Greater cost effectiveness In the long run. 

• Ability to allow staffing levels to fluctuate In response to varying 
resource needs. 

Outside evaluations may be most appropriate In situations where 
organizational tensions or mistrust call for an evaluation with maximum 
separation from the JTPA system. For example, outside consultants may 
provide greater credibility when the evaluation calls for a more 
subjective assessment of process or Implementation factors. In such a 
case, service providers, SDA staff and other stakeholders may more 
easily trust and accept the Interpretive evaluation results of an 
outsider. 

Co!!<promise Staffing Strategies 

A compromise staffing strategy Involves the judicious use of 
consultants at critical planning and Implementation Junctures of the 
evaluation where expertise Is most needed. For example, a consultant 
might be brought In solely to assess the evaluabllltv of a program (see 
Chapter 3) or to develop the evaluation design which others may carry 
out. Alternatively, a consultant's role might be strictly advisory, 
limited to reviewing and commenting on the in-house evaluation work in 
progress. In this manner, quality control might be assured, while 
consultants • fees are contained. When a formal review is conducted by 
a completely independent party, the process may be considered an 
evaluation audit , as described below. 
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Incorporating Audit Procedures into the Evaluation 
An in-between staffing solution is to supplement in-house evaluation 
activities with external auditing of those activities. In essence, the 
audit constitutes an evaluation of the evaluation, a process sometimes 
referred to as "meta-evaluation.** 



This kind of audit by an Independent third party serves several 
functions. An auditor can formally review and critique not only the 
evaluation plan, but also Implementation procedures and the final 
evaluation report. By reviewing the plan before evaluation commences, 
the auditor can spot problems, gaps and weaknesses In the plan and 
suggest changes to Improve the scientific soundness, the organizational 
effectiveness, or the efficiency of the evaluation. Using an outside 
auditor not only can Improve the utility and appropriateness of the 
evaluation, but also can enhance the credibility of an effort planned 
and executed by In-house staff. Because using an auditor offers many 
of the protections of contracting out an evaluation but at a much 
reduced cost. It Is an attractive staffing alternative. 

Audit Criteria 

If an auditor Is to be used, his or her contract should specify, among 
other things, the timing and manner In which the audit will be carried 
out, the evaluation elements to be examined and how findings will be 
presented. (For more on selecting a consultant, see the last section 
of this chapter.) The specific criteria for evaluating an evaluation 
will obviously vary with the Individual setting, but need not be 
confined only to considerations of research approach and methodology. 
The evaluation's soundness may also be Judged in terms of its 
organizational appropriateness, utility, and cost-effectiveness. On 
the following page, a checklist of meta-evaluation criteria adapted 
from Stufflebeam (1974) Illustrates the critical breadth an evaluation 
audit may entail. 



FINDING C0NSULT7U»TS : WHAT ARE THE OPTIONS? 

Choice of consultants is not limited to the few listings found In the 
Yellow Pages directory, or to RFP respondents. However, finding other 
consultant options will entail some initial effort in stepping outside 
familiar agency territory to ferret out new institutional contacts both 
in the public and private sector. 

Finding a Consultant 

While options for outside assistance will be different in each setting, 
the 11st below summarizes some of the basic kinds of external resources 
available to a local JTPA evaluation development effort. 

130 

er|c 



CRITERIA FOR AUDITING AN EVALUATION 



CRITERIA FOR TECHNICAL ADEQUACY 

• Internal validity - whether the findings are accurate 

• External validity - the extent to which the Information Is 

"generallzable" (I.e., the range of persons and conditions to 
which the findings can be applied) 

• Reliability - whether the data are accurate 

• Ob.lectlvltv - whether the data are likely to be Interpreted 
similarly by different competent judges 



CRITERIA FOR UTILITY 



• Relevance - whether the findings relate to the purposes of the 
program 

• Importance - whether the evaluation covers the most essential 
features of the program 

• Scope - whether the evaluation addresses all of the Important 
questions 

• Credibility - whether the audience trusts the evaluators and 
supposes them to be free of bias In conducting the evaluation 

• Timeliness - whether the evaluation findings are available In 
time to be used In making decisions 

• Pervasiveness - whether the findings are disseminated to all 
Intended audiences 



CRITERION FOR COST-EFFECTIVENESS 

• Cost-effec tiveness - whether the evaluation costs are kept as low 
as possible without sacrificing quality 



From: Daniel L. Stufflebeam, "Meta-Evaluatlon. " Occasional Paper No 3, 
Kalamazoo, HI: Evaluation Cenver, Western Michigan University, 1974. 
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Universities and Colleges! Both faculty and students may have 
specialized expertise they would like to lend to an applied 
evaluation research setting. while academic institutions 
rarely have a specialized degree program in evaluation, many 
departments, such as business administration, planning, public 
affairs, economics, sociology, political science, and social 
work will house individuals with an expertise in evaluation 
research. Not only faculty, but graduate students under 
faculty supervision might be able to offer valuable expertise. 
A possible constraint to using faculty and graduate students is 
the limited time they might have to devote to outside 
consulting and research. On the other hand, faculty are often 
better trained for specialized evaluation requirements and are 
often less costly as a staffing alternative. 

Research Institutes ; Even if the research institute itself 

does not have appropriate specialists, institute personnel may 
be plugged into a broader network of researchers which include 
the right kinds of specialists for a particular JTPA evaluation 
effort. 

Professional Groups; Evaluation research encompasses a number 
of professional associations. Organizations such as the 
Evaluation Research Society (a national professional 
association for evaluctors) or the American Sociological 
Association can be of use in locating qualified evaluators 
within a given area. Some states are also actively tapping 
such associations for assistance in doing JTPA evaluation 
planning; SOAs might also utilize such assistance. 

r 

Local Government! City and county planning staff (non-JTPA) 
with experience in CETA and other training and development 
programs may also be able to consult or advise for a JTPA 
evaluation project. Alternatively, these groups may offer 
important perspectives on available private evaluation 
consultants. 

business and Labo r-Affiliated Organizations ! Many such 

organizations are also keenly interested in JTPA and may have 
staff or other contacts interested in participating In an 
evaluation effort. The National Alliance for Business (NAB), 
for example, has been directly involved in the staffing of 
local JTPA evaluations. The labor-affiliated Human Resources 
Development Institute (HROI) has also been active in JTPA 
planning and assessment issues, particularly in the Title III 
programs . 



In exploring any of these options, the key is developing ongoing 
contacts within the network of researchers affiliated with these groups 
to maximize the chances of finding the right ki'^d of consultants at the 
right price. Many times, consultant resources through these groups are 
available at a much reduced cost or additional organizational resources 
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are at the consultants's command. (Refer to Chapter 9, "Funding a JTPA 
Evaluation," for more on utilizing outside resources to support 
evaluation. ) 

Selecting a Consultant 

Consultants' fees vary tremendously and so do the quality and types of 
services offered. There Is no fool-proof method for guaranteeing an 
appropriate and quality product from a hired consultant (although 
controlling the purse strings helps). However, some preliminary 
assessment (even though It may be Irrltatlngly time-consuming) of the 
consultant and the consultant services offered will Increase the 
chances of choosing wisely. Preliminary assessment might Involve: 

• Reviewing consultant's resume and written products 

^ As a first requirement, does the consultant hd^'^ the 
requisite specialized research skills and training nt^essary 
to carry out the particular activities needed? 

^ Do the products have clarity and depth? 

^ Are materials well-written, understandable? 

Do products suggest the consultant has skills and experience 
applicable to the task at hand? 

• Interviewing the consultant 

^ What are the consultant's areas of expertise and training? 

^ What are his or her conceptual or methodological biases? 

Does his or her approach to evaluation fit with your 
particular program's needs? 

How sensitive Is the evaluator to organizational factors 
affecting evaluation? 

How will the evaluator fit In? How well will he or she 
relate to others on the evaluation team or In the program? 
How Independent will he or she be? 

• Contacting previous contractors 

^ How timely have previous efforts been? 
^ What Is the quality of previous work? 

How well did the contractor work with others? 
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* Were any problems or difficulties encountered? 



• Requesting a written plan of action (Works best if evaluation 
prioritie- Hav^ already been established and the role of the 
evaluator withiri the overall framework of the evaluation is 
fairly well-defined.) 

* How well does the evaluator grasp his or her role? 

' How creatively ddei the evaluator deal with limitations and 
constraints? 

What special resources can the evaluator marshal from 
outside? (fei^.i dccess to cdimputer use, word phbcessing. 
other corisLiltants.) 

dbntract Cdricerris 

Tlie Hhal step in sfeleciirig a consultant is ironing oUt a contract that 
both parties will be satisfied with. A good contract anticipates areas 
of potential ambiguity ot conflict and protects both the consultant and 
the cblitracting a^ehcif, Aniohg other things, the contract should: 



• Specify all interim and final products and a tirhetable for each 
product's cbmpletibh. Requirements fbr iriteritn products are 
especially iiiipbrtarit in a large or lengthy project to keep the 
project on track Snd to allow for review, febmriient and revision. 

• Detail specific rbles, responsibilitiesi lines of authority and 
decision-making procedures in the evaluation project. 

• Define which resources (such as secretafial and other staff 
computer time and copying machines) the evaluator will have 
access to, and in what wayis such access will be delimited: 

• Include any follow-up responsibilities the consultant might have 
once the evaluation is complete, such as making in-person 
presentations of the findings to specified groups. 

• Determine what proprietary rights the consultant has in the 
evaluation findings or products. 

• Determine what kind of confidentiality requirements the 
consultant must agree to observe. 

• Hake payment conditibnal on satisfactory interim and final 
products, specifying (as clearly as is pbssibie) what 
constitutes "satisfactory" and through what process the 
acceptability of a product will be determined. 

• Outline expectations and a timetable for revision work. 

• Include a termination clause allowing either party to terminate 
the contract with proper advance written notice. 
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CONCLUSION 

There 1s no magic formula for staffing an evaluation effort. In 
choosing a particular staffing configuration, so many factors enter In: 
the level of 1n~house talent and expertise, staff availability, 
comparative costs of different staff choices, credibility factors and 
other political considerations, to mention a few. As the last sections 
Imply, finding and selecting a consultant to complement evaluation 
activities tacks on additional time costs In Interviewing, assessing 
consultant products and past performance, and assembling and 
negotiating a contract. Given all these staffing considerations and 
concerns, each SDA must determine what evaluation staffing pattern Is 
most efficient and feasible. Hopefully this chapter has offered some 
useful guidelines In making this determination. 



VOLUME CONCLUSION 



This guide has f ocused on various planning and Implementation Issues 
wh^ch win likely confront a local-level JTPA evaluation effort. A 
primary goal has been to help JTPA practitioners anticipate the kinds 
of planning and resource commitments an evaluation might entail. A 
theme which threads throughout the various planning steps described Is 
the Importance of the organizational context to the evaluation 
process . Initial planning effort must be devoted to assessing the 
organizational supports and constraints to evaluation and developing 
evaluation strategies which are responsive to this organizational 
framework. Various stakeholders within JTPA must be brought Into the 
planning process early to nurture their Involvement and commitment to 
the undertaking and to Insure greater relevance and util Ity of the 
evaluation findings. In the planning stages, evaluation staff may play 
a key role In bringing together diverse actors within JTPA and creating 
new patterns of communication and cooperation. 

While pointing out potential Issues and problem areas, this guide's 
central message to JTPA practitioners Is one of encouragement In the 
evaluation undertaking. Evaluation can make a difference to JTPA 
managers and policy-makers needing specialized Information about how 
efficiently and effectively JTPA goals are being met. And evaluation 
can result In Indirect organizational benefits, such as enhanced 
credibility, Improved organizational structure, or more efficient and 
accurate data collection. 

Before committing to evaluation, JTPA decision-makers not only want to 
be certain of Its returns; they also need to know that the entity will 
have the capabilities for successfully carrying out the endeavor. For 
this reason, the guide has given added emphasis to specific 
Implementation concerns. An underlying premise throughout 1s that 
local JTPA organizations, despite Internal JTPA funding restriction, 
have a number of options open to them In organizing, staffing, and 
funding an evaluation. In exercising these options, JTPA staff may 
make valuable new connections with other governmental agencies , 
universities and colleges, and private sector organizations. 

ERIC 



REFERENCES 




SELECTED REFERENCES 



ON PROGRAM EVALUATION 



GENERAL PROGRAM EVALUATION 

Alk1n, Marvin C; Dalllak, Richard; and White, Peter, using 

evaluations; Does Evaluation Make a Difference? Beverly Hills 
CA: Sage Publications, 1979. 

Chellmsky, Eleanor, ed. Program Evaluation: Patterns and 

Directions. Washington, D.C.: The American Society for Public 
Administration, 1985. 

Hatry, Harry, Winnie, Richard; and Fisk, Donald. Practical Program 

Evaluation for state and Local Government Officials. 
Washington, D.C.: The Urban Institute, 1973. 

Kimmel, Wayne A. Putting Program Evaluation In Perspective for state 
and Local Government. Human Services Monograph No. 18 
Rockvllle, HD: Project SHARE Corporation. April, 1981. 

Rutman, Leonard. Evaluation Research Methods: A Basic Guide. 
Beverly Hills, CA: Sage Publ Icatalons , 1984. 

Streunlng, Elmer L,, and Guttentag, Harcia, eds. Handbook of 

Evaluation Research. Beverly Hills, CA: Sage Publications, 



Weiss, Carol H. Evaluating Action Programs. Boston, MA: Allyn and 
Bacon, Inc., 1972. 

Weiss, Carol H. Evaluation Research: Methods of Assessing Program 

Atfectlveness . Englewood Cliffs, NJ: Prentice-Hall, Inc., 1972. 

Wholey, Joseph S. Evaluation: promise and Performance. Washington 
D.C.: The Urban Institute, 1977. 



EVALUATION PLANNING AND IMPLEMENTATION 

Adams, Kay A. and Walker, Jerry P. improving the Accountability of 

Career Education Programs: Evaluation Guidelines and 
Checklists. Columbus, OH: The National Center for Research In 
Vocational Education, Ohio State University, 1979. 

Austin, Michael J. and Associates. Evaluating Your Agency's programs. 
Beverly Hills, CA: Sage Publications, 1982. 

Alklns, Marvin C. and Associates, a Guide For Evaluation Decision 
Makers. Beverly Hills, CA: Sage Publications, 1985. 



140 

% An 



Fink, Arlene and Kosecoff, Jaquellne. Evaluation Basics: A 

Practitioner' s Manual. Beverly Hills , CA: Sage Publications, 
1982. 

Pitz, Gordon, and HcKillip, Jack. Decision Analysis for Program 
Evaluators. Beverly Hills, CA: Sage Publications, 19B_. 

Rossi, Peter H., and Freeman, Howard E. Evaluation: A systematic 
Approach. Beverly Hills, CA: Sage Publications, 1982. 

Stuff lebeam, Daniel L. "Heta Evaluation." Occasional Paper No. 3. 

Kalamazoo, HI: Evaluation Center at Western Michigan University, 
1974. 



EVALUABILITY AND EVALUATION UTILITY 

Davis, Howard R. and Salasin, Susan E. "The Utilization of 

Evaluation." In Handbook of Evaluation Research. ELuer L. 
Struening and Harcia Guttentag, eds., Beverly Hills, CA: Sage 
Publications, 1975. 

Patton, Michael Quinn. utilization^Focused Evaluation. Beverly 
Hills, CA: Sage Publications, 197B. 

Rutman, Leonard. Planning useful Evaluations. Beverly Hills, CA: 
Sage Publications, 19B0. 

Schmidt, Richard E.; Scanlon, John; and Bell, James B. Evaluability 

Asses smen t : Making Publ 1 c Programs Work Be t ter . Washington, 
D.C.: The Urban Institute. 197B. Also in the Human Services 
Monograph Series, No. 14. , Rockville, Md. : Project SHARE, 
November, 1979. 



THE ORGANIZATIONAL CONTEXT OF EVALUATION 

Bricknell, Henry M. The influence of External Political Factors on 

the Role and Me thodol ogy of Uva luation. Portland, OR: 
Northwest Educational Laboratory, 1975. 

Bardoll, Gerald L. "Type III Evaluation: Consultation and Consensus." 
Public Administration Review. Vol. 40, No. 2. March/April, 
19B0 pp. 174 179. 

Polivka, Larry and Stryker, Laurey T. "Program Evaluation and the 

Policy Process in State Government: An Effective Linkage. " 
Public Administration Review. Vol . 43, No. 3. Hay/June, 
19B3. pp. 255-259. 

Rodman, Hyman and Kolodny, Ralph. "Organizational Strains in the 

Research-Practitioner Relationship." In Readings in Evaluation 
Research. Francis G. Caro, editor. New York, NY: Russel Sage 
Foundation, 1977. 



141 



Weiss, Carol H., and Barton, Allen H. Making Bureacracles work. 
Beverly Hills, CA: Sage Publications, 1980. 

Wlldavsky, Aaron. "The Self-Evaluating Organization." Public 

Admlnlf^tratlon Pevlew. Vol. 32, No. 5. September/October, 
1972, pp. 509-520. 



EVALUATION COSTS 

Alkin, Marvin C, and Solmon, Lewis C. The costs o£ Evaluation. 
Beverly Hills, CA: Sage Publications, 1983. 



FUND SEARCH RESOURCES 

The Foundation Center publishes extensive mattrlals on foundations and 
corporate donations activities (888 Sev^ntl, Avenue, New York, NY 
10106). Some of the main reference publications are described briefly 
below: 

The Foundation Directory 

This lists over 2,500 foundations, providing a summary of each 
foundation's purposes, award amounds, officers, and application 
procedures. It Is revised annually, and has mid-year supplements. 

Foundation Grants Index 

This lists over 10,000 grants awarded by American philanthropic 
foundations. It Is upda^;ed periodically, not necesaarlly annually. 

Foundation Center Source Book Profiles 

This covers only the largest foundations In the United States. It 
provides a comprehensive survey of Information on these organizations. 
Including grant awards and current programs. It Is periodically updated. 

National Data Book 

This lists over 20,000 organizations which have been classified by the 
IRS as private foundations. Volume II Is a state listing of 
foundations In rank order by descending assets. 

Comsearch Printouts 

These are computer printouts In approximately 55 subject areas, listing 
grants by more than 400 major foundations. The data base Is the other 
Foundation Center books. These can alkso be obtained In microfiche. 

Corporate Foundation Profiles 

This Is a comprehensive analysis of each of over 200 of the largest 
company-sponsored foundations. 

Grant Writing Materials 

In addition to the above directories, the Foundation Center also has 
numerous how-to materials on planning, preparing, and submitting grant 
proposals. 



EKLC 



142 142 



other directories focusing on corporate donation activities Include: 

Corporate SOO. San Francisco: Public Management Instltue (Annual), 

This annual directory tracks corporate giving of the largest 
corporations. Quarterly updates are also available, 

Taft Corporate Directory. Washington, D,C,: The Taft Corporation 
(monthly). 

Corporate Giving, Washington, D.C: The Taft Corporation (monthly). 
Corporate Updates. Washington, D.C: The Taft Corporation (monthly). 



EKLC 



143143 



Appendix 



A Report on a National/State Survey of 
Local JTPA ConstituencBes 



By Bonnie Snedeker 

Snedeker Scientific, Inc. 



TABLE OF CONTENTS 



INTRODUCTION 
METHODOLOGY 

OVERVIEW OF LOCAL CLIMATE FOR EVALUATION 
LOCAL USES OF EVALUATION INFORMATION 
PRIORITIES FOR OUTCOMES MEASUREMENT 
PRIORITIES FOR EVALUATION DESIGN PACKAGES 
SUGGESTIONS FOR EVALUATION DESIGNERS 
LOCAL CAPABILITIES AND CONTINGENCIES 



Us 



INTRODUCTION 



This report summarizes the findings of a telephone 
survey on the JTPA evaluation priorities and 
capabilities of local -level constituencies. The survey 
was conducted by Snedeker Scientific, Inc. and the 
Seattle-King County Private Industry Council during 
February through April, 1985. Included in thi. survey 
were SDA, PIC, and local government representatives 
from 24 SDAs - 12 in Washington State and 12 from a 
specially selected national sample . 

The survey is part of the JTPA Evaluation Design 
Project, conducted by the Washington State Employment 
Security Department, with support from the National 
Commission on Employment Policy and the I.B.M. 
Corporation. This project, through the combined 
efforts of a team of evaluation design specialists and 
a national advisory committee of state and local 
practitioners, will develop and produce a series of 
evaluation models which can be used to assess the 
effectiveness and impact of JTPA programs arid systems 
at the state and SDA levels. 

The purpose of the survey is to provide input from 
local constituencies which can help the evaluation 
designers to develop models appropriate for JTPA 
evaluation at the local as well as the state level. 
Specifically, the survey sought useable information in 
the following areas: 1) the climate for evaluation 
initiatives at the SDA level; 2) local priorities and 
needs for the use of evaluation information ; 3) local 
issues and priorities in regard to specific evaluation 
measures and design capabilities; 4) local suggestions 
for evaluation designers; and 5) local system 
capabilities and contingencies. 

The survey was designed to produce maximum input from 
local constituencies within some fairly tight resource 
and time constraints. We make no claims of statistical 
significance for its findings. It is best viewed as a 
practical research effort, intended to tap local 
constituencies on a selective basis and produce 
descriptive and qualitative information of particular 
interest and utility for evaluation designers. It is 
hoped that these findings will be of interest also to 
those who participated in the survey, as well as others 
concerned with JTPA evsluation at the SDA level. 
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This report summarizes the findings of a telephone 
survey on the JTPA evaluation priorities and 
capabilities of local-level constituencies. The survey 
was conducted by Snedeker Scientific, Inc. and the 
Seattle-King County Private Industry Council during 
February through April, 1985. Included in thi. survey 
were SDA, PIC, and local government representatives 
from 24 SDAs - 12 in Washington State and 12 from a 
specially selected national sample . 

The survey is part of the JTPA Evaluation Design 
♦ Project, conducted by the Washington State Employment 

Security Department , with support from the National 
Commission on Employment Policy and the I.B.M. 
Corporation . This project , through the combined 
efforts of a team of evaluation design specialists and 
a national advisory committee of state and local 
practitioners, will develop and produce a series of 
evaluation models which can be used to assess the 
effectiveness and impact of JTPA programs arid systems 
at the state and SDA levels. 

The purpose of the survey is to provide input from 
local constituencies which can help the evaluation 
designers to develop models appropriate for JTPA 
evaluation at the local as well as the state level. 
Specifically, the survey sought useable information in 
the following areas: 1) the climate for evaluation 
initiatives at the SDA level; 2) local priorities and 
needs for the use of evaluation information; 3) local 
issues and priorities in regard to specific evaluation 
measures and design capabilities; 4) local suggestions 
for evaluation designers; and 5) local system 
capabilities and contingencies. 

The survey was designed to produce maximum input from 
local constituencies within some fairly tight resource 
and time constraints. We make no claims of statistical 
significance for its findings. It is best viewed as a 
practical research effort, intended to tap local 
constituencies on a selective basis and produce 
descriptive and qualitative information of particular 
interest and utility for evaluation designers. It is 
hoped that these findings will be of interest also to 
those who participated in the survey, as well as others 
concerned with JTPA evaluation at the SDA level. 



EKLC 



PAGE 44 6 



METHODOLOGY 



The survey approach was designed by Bonnie Snedeker 
(Snedeker Scientific Inc.) and Brian 0' Sullivan 
(Seattle-King County PIC) under the direction of the 
Project Coordinator, Ann Blalock (Washington State 
Employment Security Department.) 



It was determined at the cutset that 
and financial resources available, the 
for tapping useful input from local 
would be a telephone survey of a limited 
selected sample of SDAs^ Upon further 
was determined that two groups of 
surveyed: 1) a national sample 
representing a structured mix of key SDA 
100 percent of the (12) SDAs in Washingt 



, given the time 
best approach 
constituencies 
but carefully 
consideration it 
SDAs would be 
of 12 SDAs, 
types; and 2) 
on State. 



The 12 SDAs in the national sample were selected 
through a process of consultation with national 
advisory committee members, state JTPA officials, 
National Alliance of Business researchers, and a 
network of other national, regional, and local 
contacts. The national sample was structured to 
include a mix of key SDA types in regard to the 
following variables: 1) geographic region of the U.S.; 

2) magnitude of JTPA funding (size of SDA II-A grant); 

3) population density (urban, suburban, rural); 4) 
jurisdictional configuration (citv, county, 
multi-county, etc.); and 5) apparent in/olvement to 
date in local-level evaluation initiatives. 

A descriptive breakdown of the national sample by four 
primary criteria is provided below 



1) Size of II-A Grant (FY 1984) 

Over $6 million 
$2-$6 million 
Under $2 million 

2) Population Density 

predominantly Urban/metro 

predominantly Suburban 

predominantly Rural 
Mixed 



2 SDAs 

3 SDAs 
7 SDAs 



4 SDAs 

2 SDAs 

3 SDAs 
3 SDAs 
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3) Jurisdictional Configuration 

City Ortly i SDA 

City/County 3 sDAs 

County or Balance of County 2 SDAs 

Multi--Couilty 5 SDAs 

Collection 6f Townships 1 SDA 

^) SbphifiticatloiiJ Indication of 
Substantial Involvement in 
Loc&l fivaluation Initiatives 

](es 6 SDAs 

^ 6 SDAs 

the Katlonal sample Includes the following SDAs: 

1. Metro-Southwest SDA, Massachusetts )Region I) 

2. Cumberland dounty SDA, Maine (Region II) 

3. ^ Balance of Onondaga County SDA, New York (Region 

A. Baltimore County SDA, Maryland (Region III) 

5. South Florida Employment and Training Consortium 
(Region IV) 

6. Gulf Coast Business Services Corporation. 
Mississippi (Region IV) 

7. Lansing Tri-County Consortium, Michigan (Region V) 

8. City of St. Paul, Minnesota (Region V) 

9. ^ Balance of Captital Planning Region, Texas (Region 

10. SDA V, Iowa (Region VII) 

(Region^VI'I)^^°^°^"^ Training Consortium, Colorado 

12. San Diego Regional Employment and Training 
Consortium, California (Region IX) 

The 12 Washington SDAs, which were included in the 
survey, range from a large metropolitan system with a 
II-A grant of nearly $8 million to a rural eight-county 
SDA with a II-A gfant of less than $1 million. In 
comparison with the national sample, Washington SDAs 



were somewhat less likely to have conducted substantial 
local evaluation initiatives (only 4 of the 19 
assessed as having substai?ia? e.^lUeTcl II Ull 
SIsMno^ ^"n" """^ multi-county SDAs in tH 

Washington sample (8 of 12), fewer predominantly urban 
or metropolitan SDAs (3 of 12), and iore SDAs lith . 
predominantly rural or mixed urban/rural populatJoI; (8 

The Washington SDAs surveyed include: 

SDA I. Olympic Consortium 

SDA II. Pacific Mountain Consortium 

SDA III. Northwest Washington 

SDA IV. Snohomish County 

SDA V. Seattle-King County 

SDA VI. Tacoma-Pierce County 

SDA VII. Southwest Washington Consortium 

SDA VIII. Pentad Consortium 

SDA IX. Tri-Valley Consortium 

SDA X. Eastern Job Training Partnership 

SDA XI. Benton, Frankin, Walla Walla Counties 

SDA XII. Spokane City and County Consortium 

variety of local constituencies, including: SDA-level 
administrative rtaff; PIC members; and local elected 
?nri ^ . °° P^^^ experience with research at the 
SDA level it was anticipated that the greatest amount 
°ffV,"^^§Sl^ information would be derived from interviews 
witn bUA administrative entity staff. It was 
determined that SDA directors and/o^ 

fourirfoJ t-h'^" P"^"'^^^ information 

source tor the survey. 

IJl-.fT^L approach included initial contact with the 
administrative director in each SDA. This «as followed 
by a scheduled telephone interview with the director or 
«^n,,^ ^?n!^" ^'fsiSnated being most knowledgeable 

about SDA-level evaluation issues. In conducting SDA 
interviews, we used structured interview 
guides 3nd reporting formats, which included a 



combination of dp^h-^nded ^ limited cJioice, arid sdaled 
rating response , iteffis. SDA f€isp6ndeht interviews 
lasted bgtvr'eert iSifiiiteS dependirig lipoh tRe time 

availability iiid intgfeSt of the respondent- and the 
^Sxtensiveri^ss of tfi^ feva'ltialtiofi issties and activities 
ciirreritly on tfie SDA dg^nda. 

We iskSd ST^k difefctdfs to fecdmmfend PiC ifi§f6feefs and 
local gieeted eifieiais witfi an interest 6t iriVSlveraent 
in ^ JTPA .^^aiiJafeibri. All SfiA respondents jifdvided 
cbntact iUfdrMildR f of at Ifeast 6ri6 PIG 6f Ideal 
§6vef fifiSfit fepii§§iii&t±i§ i A Short ef interview guide 
fiind repof tih§ ^fgfitJsit^ which dhpiicatfefd gSnie of tfie 
ite'ms 8n tR§ SfiA ifistfuflferit , was used lii ddndiifcting tfie 
pig/Leo inter vi§wS; the&e Interviews tygicdlly were 
fitccompiisfied within 30 mintifcesi The gehfefal level of 
SwafefiSsS afifl dBiiity to addfeSfe spfecific 6valuStidn 
Issues Has, tifidfef Sfcandably ^ Ibwef &Bdtig the PlG/LEO 
respondents ; 

Iiiterviei? rejiSfting fdfmats weffe cdfijpi^tfed for a total 
of 49 iiidiVidtlais, iricludihg: 12 Washington SDA 
respondents; 12 SDA fe&^dhdefits ffdfn the national 
sample; iS Pi§ afembfeffsj ihd 16 Ibcal elefcted officials. 

Recordeid responses were analyzed, by subject area and 
on an item-by-itetii feasis, for each bf the major 
categories of respondents and §6i"b&s all respondents* 
Open-ended responses were analyzed §nd grouped by 
frequency; multiple-choice resiiohses wefe tabulated; 
and mean ratings were calcultifced for scaled rating 
items • 

In capturing, analyzing, and rejibrting telephone survey 
information, resfearcher accuraejr# understanding, and 
interpretation are dbvibiisly dpeh to question. While 
recognizing our own limitatibfifej we take fiill 
responsibility fir^r the material contained in this 
report. 
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OVERVIEW OF THE LOCAL CLIMATE FOR EVALUATION 



It is the goal of this project to produce evaluation 
design materials that will stimulate, guide, and 
support JTPA evaluation efforts - et the lucal SDA 
level, as wall as at the state level. 

Receptivity for the products this ])roject irs developing 
cannot be taker, for grantea. Some of the factors that 
will detei.mine whether our products are usee? at the SDA 
level are: 



1) The extent to which 
interested in evaluation 
evaluation initiatives; 



IoclI constituu..ji6s are 
issues and willing to pursue 



2) The extent to which local systems are capable of 
supporting evaluation initiatives; 

3) The extent to which there are felt needs for outside 
assistance in conceptualizing, designing and 
structuring local evaluation initiatives; 

4) The extent to which the products we develop fit - 
or can be fitted to - the particular priorities, needs 
and capabilities of local constituencies. 

This survey found considerable interest in evaluation 
issues among local constituencies. It also found 
considerable variation in hov these issues have been 
conceptualized and acted upon to date at the SDA level. 

One trend was strong across the survey sample . LocbI 
accountability for performance is a key feature of the 
JTPA environment. Virtually all of the respondents 
reported that tracking and monitoring program 
performance is a high priority for their SDA. At a 
base level, all local constituencies expressed a 
concern for achieving and documentiw»g job placements 
(and other positive outcomes) for participants at a 
reasonable cost, and all of the SDAs have some system 
for basic performance tracking. 

In most SDAs, concern for evaluation extends beyond 
^^erf ormance tracking and documentation. The local 
constituencies we surveyed ate interested in capturing 
feedback - both quantitative and qualitative - on 
program performance and effects; anal^.yzing this 
feedback, extracting useful findings, and applying them 
in a variety of .conte;.:ts. 

152 



PAGE 9 



This analysis, and utilization of program information 
is conducted by SDAs at varying levels of formality, 
sophistication, and consistency. Such activity is often 
geared toward immediate funding or design decisions or 
ongoing management efforts which may draw on other 
types of information, such as labor market and 
demographic data, national R&D findings, as well as 
locally generated feedback on program operations and 
outcomes. SDAs are more likely to classify these 
activities as analysis, assessment, or even planning, 
than as "evaluation". 

For many local-level actors, the term evaluation 
connotes a formal study or structured review of the 
systom at large or of specific programs or components, 
conducted at a specific point in time, by persons 
outside of the ongoing management and planning 
efforts. This type of evaluation, while viewed as 
worthwhile and potentially useful, has been less 
frequently employed among the SDAs surveyed. 

Definitional and conceptual confusion made it somewhat 
difficult to gauge quickly the extent to which each SDA 
has actually engaged in local evaluation initiatives to 
date. Upon initial inquiry, one-half of the national 
sample and nearly two-thirds of the Washington SDAs 
reported that they had not yet conducted substantial 
evaluations under JTPA. On further inquiry, it became 
apparent, however, that virtually all of the SDAs were 
capturing, analyzing, and using certain types of 
performance data on, at least, an ad hoc basis. The 
other SDAs - half of the national sample and one-third 
of the Washington systems - were able to describe 
specific evaluation activities which had been locally 
designed and initiated. 

Much of the more developed evaluation activity we 
encountered was found among the larger SDAs in the 
sample. When it comes to local capabilities for 
supporting evaluation initiatives , larger SDAs have 
some distinct advantages. These include: greater 
financial resources and more flexibility in 
administrative budgeting; more staff, including staff 
with needed technical expertise; a larger data base; 
and, in some cases, more sophisticated data retrieval 
and analysis capabilities. However , we also 
encountered a number of smaller SDAs, including some 
with II-A allocations of under $1 million, which had 
found creative ways to carry out locally tailored 
evaluation initiatives. 

Even with current limitations on resources and less 
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than optimal capabilities, the majority of SDA systems 
have the basic capabilities, interest, and desire to do 
more in the area of evaluation. Over 80 percent of the 
SDAs surveyed have automated data storage and retrieval 
systems which are capable of, or can be adapted to 
support increased evaluation efforts. Over 70 percent 
have some kind of system in place for capturing 
follow-up information on post-program (13 weeks or 
longer) outcomes. and 75 percent indicated specific 
plans to upgrade or expand their evaluation efforts 
during the coming year. 

Major areas of development in local-level evaluation 
include : 

1) Instituting longer-term follow-up and/or expanding 
or enriching follow-up contacts with employers or 
participants; 

2) Conducting more systematic and detailed analyses and 
making greater use of program and follow-up data; 

3) Upgrading the MIS or implementing a new information 
system; 

A) Conducting process evaluations or special assessment 
studies ; 

5) Creating new linkages for accessing and using 
ncn-JTPA data bases (welf are/U.I . ) 

Finally, our survey found that local constituencies do 
recognize a need for outside assistance and guidance on 
evaluation designs and techniques. This need was 
expressed particularly by smaller systems which lack 
the expertise or resources to develop tailor made 
approaches from the ground-up. Nearly 30 percent of 
the SDAs surveyed had solicitsd some outside assistance 
in designing or conducting evaluations. Even more 
sophisticated, larger systems which are confident of 
their own technical expertise expressed a desire for 
high quality "off-the-shelf" designs and technical 
assistance guides which could be adapted to local 
purposes . 

This does not mean, however that local constituencies 
are likely to jump into implementing any model that 
might be offered. The bottomline criteria for 
acceptable designs are simplicity, practicality, and 
reasonable prospects of producing outputs tailored to 
local-level uses. SDAs already feel burdened by 
state-imposed reporting and follow-up systems which are 
not geared to readily produce the types of information 
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that would be most useful for local purposes. 



Local-level needs for evaluation information and 
analyses are different from national or state-level 
needs, because local constituencies are directly 
Involved in managing and awarding contracts, developing 
policy frameworks, allocating resources, and desicninR 
and delivering services. 

The next section of this report looks at local 
priorities and needs in regard to specific uses of 
evaluation information. 
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LOCAL USE OF EVALUATION INFORMATION 

1) Performance Management and Corrective Action 
Survey responses indicate that the most common use to 
date of program performance data generated at the 
local level has been in the ongoing management of 
program operations and contracts, the identification of 
problems, and the initiation of corrective action 
efforts. Identifying performance problems and taking 
prompt action to correct them was cited as the most 
important current use of evaluation information by SDA 
directors and staff. (PIC members and LEO respondents 
tended to rate this use as slightly less important.) 

The thrust of ongoing management efforts at the SDA 
level tends to be double-pronged: 1) to ensure that 
actual levels of participant end financial transactions 
are synchronized with levels specified in the job 
training plan and individual contracts; and 2) to 
ensure that the system as a whole will meet or exceed 
itB annual performance stand^^rds. 

Most SDAs prepare monthly managerial reports which 
analyze , at minimum, actual expenditures, enrollments , 
participant characteristics, and termination outcomes 
in comparison with planned levels - for the program 
overall and for each major training component and/or 
contractor. Performance data is summarized in 
quarterly reports which measure system-wide performance 
with regard to each of the basic performance standard 
indicators . 

In the review of program performance data for ongoing 
systems management, particular attention is paid to the 
number and rate of terminees entering employment. 
Placement performance is fairly easy to track and 
analyze at any point in time on a contractor or 
component basis. Managerial action is taken promptly 
when placements are lagging seriously behind planned 
levels. 

Controlling costs is another key management concern. 
Computing and tracking cost per entered employment (or 
cost per positive termination for youth) on an ongoing 
basis presents some difficulties for most SDAs. 
Financial data is not integrated with participant-based 
(MIS) data in the large majority of SDA systems. Cost 
data sometimes lags far behind information on 
participant transactions. Some SDAs only compute unit 
cost measures on a quarterly or year- end basis. Unit 
costs are often controlled through ceilings written 
into cost reimbursable contracts or (more effectively) 
through the provisions of unit price contracts, which 
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pay a fixed amount per placement. Larger SDAs with 
fixed unit price contracting systems generally maintain 
separate cost accounting/verification systems to track 
on a weekly basis contractor payments linked with 
specific participant transactions. It is the 
exceptional SDA (only 2 of the 24 surveyed) that has a 
fully integrated financial and participant-based 
information system. 

Computing actual versus planned performance statistics 
for basic indicators is less difficult for many SDAs 
than analyzing on an ongoing basis how actual 
performance stacks up against state-applied performance 
standards. State use of the regression model in 
adjusting performance standards for each SDA means that 
performance standards may be considerably altered by 
unanticipated changes in the program. For example^ a 
shift in the demographic composition of enrollees or in 
the average duration of enrollment will alter the cost 
per entered employment standard against which the SDA 
will be evaluated by the state. Ii order to calculate 
how actual performance compares with the performance 
standard at a given point in time, an SDA must be able 
to accurately compute both actual unit costs and the 
adjusted performance standard. Many SDAs lack this 
capability at the current time. They concentrate ^ 
instead, on managing performance to correspond as 
closely as possible with the plan and wait until the 
end of the year to see how they will "come out" in 
regard to the cost per entered employment and cost per 
positive youth termination performance standards. 

Another important focus of program management is 
monitoring performance in regard to key target groups. 
In some SDAs , concern is limited to groups for which 
specific enrollment levels have been mandated by the 
state (such as dropouts or WIN registrants.) But at 
least half of the SDAs surveyed have placed a high 
priority on achieving at least specific enrollment 
levels for other "most in need" target groups (such as 
minorities, single parents^ and persons with 
handicaps.) With the exception of welfare recipients 
(for whom the entered employment rate is a mandated 
performance standard), management analysis of target 
group performance is typically limited to enrollment, 
as opposed to termination, analysis. About half of the 
SDAs reported that corrective action procedures are 
initiated when program contractors fail to meet target 
group enrollment objectives. 

Other key performance indicators in program management 
are the average wage at placement and the percentage of 
placements that are training related. At least 
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one-third of the SDAs surveyed have set specific 
objectives in one or both of these areas for major 
program components or individual training projects. 
These objectives are based in part on a detailed 
analysis of actual placement data. 

Job retention is an increasing concern in performance 
management at the SDA-level. Most of the SDA systems 
included in the survey track 30 day retention data, and 
at least one-quarter of the SDAs pay explicit attention 
to this indicator in ongoing program management. Only 
a very few SDAs, however, incorporate longer-term 
follow-up data in managerial performance reviews. 

2) Input for Local Policy Development 

Survey responses indicate that the usefulness of 
evaluation information in orienting, educating. and 
informing local policymakers has been second in 
importance only to its usefulness in ongoing systems 
management. As a group, PIC members tended to consider 
this use of evaluation information to be the most 
important use to date at the SDA-level. In most SDAs, 
the PIC bears the primary responsibility for 
establishing local policy goals and program objectives. 

The establishment of a meaningful SDA policy framework 
requires a good understanding of the local program 
environment. At a minimum, PIC policymakers have been 
required to understand the national performance 
standard indicators and determine the extent to which 
JTPA programs in the SDA will be required to meet 
specific performance goals. An analysis of first year 
performance results has provided baseline information 
for setting specific program objectives for subsequent 
years in most of the SDAs surveyed. 

In some SDAs - perhaps half of those surveyed - local 
policy development is limited to the goal of exceeding 
all etate-levied performance standards, thereby 
demonstrating superior performance in comparison with 
other SDAs in the state. 

While SDAs with more developed policy frameworks also 
take national/state performance standards into account, 
their policy goals tend to reflect local circumstances 
and preferences. Several of the SDAs surveyed have mads 
explicit policy choices to place greater priority on 
providing high quality training in specific 
occupational areas or on serving specified "most in 
need*^ segments of the eligible population than on 
achieving the cost per entered employment standard 
specified by the state. Some PICs are setting 
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stringent objectives for target group enrollment, 
training-related placement rates, retention rates, 
average wage at placement, or ceilings on the 
proportion of direct placements which will be allowed 
These objectives are designed to reinforce local policy 
orientations • 

The base of information and understanding required to 
support the development of a local policy framework 
goes beyond exposure to statistical summaries of 
enrollments , costs, and termination outcomes . Analysis 
of information not specifically generated through 
program activity (such as demographic and labor market 
data or national R&D findings) clearly plays a role in 
framing a local policy orientation. But perhaps more 
important is a substantive assessment of local program 
design and effectiveness with regard to key target 
populations and targeted occupations or sectors of the 
labor market* PIC members and SDA staff report that 
process evaluations, case studies, and other assessment 
efforts that provide descriptive and qualitative 
feedback as well as quantitative analysis have been 
extremely valuable in giving policymakers a clear 
picture of who the programs are serving, what they are 
actually doing, and what outcomes and impacts they can 
be expected to achieve. 



3) Program Funding and Design 

Local constituencies make decisions on how SDA 
resources will be allocated or deployed and on what 
programs, services, or contractors will be funded. In 
some cases SDA-level roles in program design and 
development are limited to setting performance goals, 
allocating funds, selecting contractors, and 
negotiating contracts. But in most cases PICs end SDA 
staff have a more extensive role in shaping program 
design. As shown by a recent survey conducted by the 
National Alliance of Business, administrative entities 
are directly involved in program operations in over 
half of all SDAs. 

It ia in the areas of program funding, design, and 
development that evaluation information is viewed by 
local constituencies as having its greatest potential 
usefulness . 

A num?;-2r of the SDAs we surveyed are using locally 
generated performance data as a basis for allocating 
training dollars across functional service areas. 
Several are using linear projection models, which use 
past performance data as a basis for predicting the 
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performance outcomes (enrollments, placements, unit 
costs, etc.) which can be expected from different 
dollar allocations to functional service categories 
(OJT, institutional skills training, job club/placement 
services, etc.) The SDA selects and uses the 
allocation formula most likely to maximize performance 
results. While more sophisticated allocation models 
may include factors such as service to key target 
populations, wage rates, and training-related outcomes 
in their performance projections, none to date have 
included longer-term employment or earning gains for 
participants or reductions in welfare payments. 

One of the simplest and most prevalent uses of 
evaluati^on information has been in making annual 
funding decisions with regard to specific contractors 
or projects. PIC committees analyze and compare past 
performance results in considering contract renewals 
and funding levels. Given the grant reductions being 
experienced by most SDAs, funding decisions often 
revolve around how necessary cutbacks will be 
distributed across current contractors or projects. 
Those training projects or program operators who do not 
compare favorably with their competitors in regard to 
job placement/positive termination or unit cost 
performance are the most likely to have their funding 
cut back or eliminated. Here again, while factors such 
as ability to serve high priority target populations or 
to provide priority services not otherwise available 
are generally considered in such funding decisions, 
infer mat ion on probable longer-term impact and 
effectiveness is typically not available to 
decision-makers . 

SDA staff and PICs become more substantially involved 
in program design through such activities as: the 
development of RFP criteria, programmatic 

specifications, contract provisions , and youth 
employment competency standards; the selection of 
occupational areas for institutional skills training; 
the development of training projects and curriculum. 

In these endeavors, local constituencies are most apt 
to be concerned with determining the types and levels 
of skills, knowledge, and behavioral traits associated 
with securing and maintaining employment (often in a 
specific sector of the job market) and with identifying 
the best training approaches and techniques for 
assisting various types of JTPA-eligibles to secure 
needed competencies and make a successful labor market 
adjustment. SDAs are also interested in identifying 
those industries, occupations, or type of employers in 
the local market which offer the best prospects for 
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employing JTPA trainees in jobs with reasonable 
entry-level wages and opportunities for stability and 
advancement • 

Many of the people who participated in our survey 
expressed the desire for local evaluation information 
that would help in determining which training 
approaches and services are most likely to be effective 
in achieving longer-term benefits for participants. At 
minimum, local constituencies would like to be able to 
assess whether substantial investments in occupational 
skills training, or other more intensive developmental 
strategies result in benefits for participants and the 
community which are appreciably greater than those 
derived from less costly, shorter-term training and 
placement services. 



A) Publicly Documenting Accomplishments 

While most of the survey respondents placed importance 
on the use of local evaluation information to publicly 
document the accomplishments of the JTPA system, local 
elected officials were the only group to give this 
utilization area priority over policy developn-ent and 
program funding and design. 

The large majority of respondents hold the belief that 
documentation and dissemination of positive evaluation 
results are important to the survival and funding of 
JTPA at the federal level. But they are not cr^rtain how 
local evaluation initiatives can effectively contribute 
to this effort. Solid justification of JTPA as an 
investment strategy requires some type of net impact 
analysis, which all of the constituencies surveyed felt 
was beyond the capabilities of local systems . 

Nonetheless , most of the respondents felt it was 
important to establish a positive image within the 
community - both in terms of establishing public 
support for JTPA as a worthwhile program and in terras 
of building a reputation for the Private Industry 
Council as an effective organization. 

While a nuraber of respondents said that they try to get 
as much public relations mileage as possible out of 
favorable program assessments or performance results , 
comparatively few felt that their SDA had been very 
effective to date in drawing public attention to JTPA 
accomplishments. In comparison with other programs, 
such as public education and CETA at its height, JTPA 
has a small resource base and typically affects only a 
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very limited segment of the community. There does not 
appear to be much community interest in JTPA in 
general, and there is even less interest or 
understanding for the types of findings typically 
generated by local assessment efforts. Descriptive 
reports of successful economic development linkages or 
special projects with a strong human interest factor 
are more apt to generate media attention. 



5) Marketing 

Survey respondents - particularly PIC members as a 
group and SDA administrators from Washington State - 
placed a relatii'ely high priority on demonstrating the 
utility and benefits of JTPA programs to local 
employers. But few SDAs to date had managed to 
effectively develop or package evaluation information 
for use in marketing the program to employers. 

PICs and SDA administrative entities have an expressed 
interest in generating broader private sector support 
for employment and training efforts targeted on 
JTPA-eligibles. But SDA-level marketing campaigns and 
materials have tended to use only generalized messages 
in alerting employers about the potential benefits of 
JTPA involvement. This is particularly true for SDAs 
which contract out all service delivery functions and 
have little or no contact with the employers who 
actually train and employ JTPA participants. 

However, a significant portion of SDA administrative 
entities (over 40 percent according to the NAB survey) 
are directly involved in the delivery of JTPA placement 
services. Our survey responses indicate that SDAs with 
this kind of service delivery role are more likely to 
carefully analyze placement data and use their findings 
as a basis for targeting training, job development, and 
placement efforts. 

Survey respondents also indicate a growing trend toward 
SDA-ini tiated contacts with . employers as a means of 
generating useful feedback on employer satisfaction and 
program effectiveness. Such contact mechanisms could 
be used to generate evaluation information tailored for 
future marketing efforts. 
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PRIORITIES FOR OUTCOMES MEASUREMENT AND ANALYSIS 



Survey responses show that in analyzing evaluative 
information for program planning , management , and 
funding, SDAs to date have relied heavily on a fairly 
limited array of short-term indicators. However, the 
survey found a growing interest at the SDA level in 
measuring and analyzing longer-terin outcomes of various 
types. A number of the SDAs are already investing 
considerable resources in capturing follow-up data. 

We asked SDA respondents to rate the level of 
importance for their SDA/PIC of information on a 
variety of possible outcomes measures. Respondents 
rated each measure proposed on a scale of 1-5 (with 
l«of no importance and 5=extremely important.) We 
compiled mean ratings on each measure for both the 
national sample, the Washington SDAs, and for the 
combined sample. Priority measures are listed below 
(with comments) in order of their composite mean 
rating . 



1 ) Job Stability - Retention of Employment with the 
Placement Employer (Mean Rating « 4.1) 

This outcome measure received the highest overall mean 
rating, but it was rated somewhat higher by Washington 
SDA respondents than by those in the national sample. 
Most SDAs view job retention as an interim rather than 
a final indicator of program outcomes. (Terminees are 
not expected to remain employees of the placement 
employers on a lifetime basis.) But retention data at 
13 weeks is becoming more readily available and is 
viewed as a highly useful indicator of training quality 
and program effectiveness 



2) Differentials in Results/Outcomes by Service 
Strategy or Proiect fMean Rating » 3.9) 

SDA respondents from the national sample were more 
likely than Washington respondents to place a high 
degree of importance on ability at the SDA level to 
measure differentials in outcomes or results across 
service strategies and projects. Most likely this is 
because of the greater number of smaller systems within 
Washington State which fund fewer individual projects, 
serve fewer participants, and tend to make less 
distinction among service strategies in program funding 
and assessment. A number of SDA respondents stressed 
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the opinion that measures of differentials in outcomes 
or results by service strategy/project must take ?nt« 
account differentials in the types' and charac^^risJjcs 
of participants served. «=ij.suics 

^) -Participant Earnin g Gains or Losses fS.R'i 

Few of the SDAs surveyed are currently able to estimate 
In" TnTull'° P^^^i'^iP-^- experience an inc^e^s^ 

comnarpS ^i ^^^^"^"8= after program termination (as 
compared to pre-program experience.) None of the SDA 
respondents was interested in conducting net-impact 
evaluation at the local level, but most would like to 

ea^nin.':"%'n ^'^^"^^^i- - participants' post-program 
earnings. In wage reporting states, U.I. data was 
viewed as the best potential source of ^ost-prog^am 
earnings data. Follow-up contact with participanJs^was 
not generally viewed as a very accurate source of 
iere"°'ii:;er earnings. Annual earnings gains 

roi^Lr./ * ^ meaningful measure than 

resSonJpT P"/P°^t hourly wages. But a number of 
Jar?Jcinn«t interested in the extent to which 

^^st^pl^c^Lnt y^fr. ^^^''^^^ ^"^^ 

Outcomes- ^'"'"^ RelatPdn.ss g f^ K.p 1oym_ent 

This measure received a higher mean ranking from the 
national sample SDAs than it did from Washington 
respondents. The extent to which post-program jobs ^are 
related to the type of training provided is of greater 

theJr"TT?A^^i%f " substantial portion of 
tneir jrPA dollars to occupational skills training 

Sar^ici'nl interested in wheJJie? 

participants remain employed in training-related 

:"eTl\lrr ZLIHV "-'"^^ occupational 

5) Reductions in V^ lfare Pnym&nt.fiC^ 'i'i 

culL.lr. "^^^ ^^^^ surveyed reported that they were 

^^vmpn^i^ estimating or measuring reductions in welfare 
\ "^l^ calculating welfare entered 

employmenj: rates. The abJ.lity to accurately calculate 
f. welfare payments depends upon access to 

' : ^^"y respondents had become 

o"^^"* 5^^^^^ attempts to gain such access. 

«rM.vo5 '^^^^ ^"'^h access would ever be 

fl±uZl t. ^^'^^^ ^^^^^ faulted the state for 

failure to secure cooparatiou frca the welfare 



PAGE 21 1B4 



department. But Ecost respondents felt that information 
on welfare payment reductions could be very useful - 
both in targeting program services and in demonstrating 
positive returns on the JTPA investment. 



PRIORITIES FOR EVALUATION DESIGN PACKAGES 



In an attempt to focus input from local constituencies 
more specifically on potential evaluational models, we 
askfcd siTvey respondents to r. te fhe level of 
usefulness or importance to their SDA of various 
elements that mi^ht be included in a total evaluation 
package . 

Their ratings (on a 1-5 scale) for each potential 
element were averaged, overall and for each cf the 
following groups: 1) national sample SDA respondents; 
2) Washington State SDA respondents; 3) PIC members! 
and 4) local elected officials. Responses are 
summarized below, in the order of priority indicated by 
overall mean ratings. 



1) Strategi es for evaluating program effectiveness in 
achieving long er-term employment and earnin g s gains fo r 
participants (Mean Ratxng = 4.4> : ' 

The need for evaluation designs which offer a 
practicable approach for SDA-level evaluation of 
program effectiveness in achieving longer-term 
employment and et:rnings gains for participants was 
rated most important by both national and Washington 
SDA respondents. PIC members gave this element a mean 
rating of 5 (on a 15 scale.) (The mean rating given 
by local elected officials ,^as only 3.7.) 

The interest of SDAs in gaining insights on longer-terrii 
program effectiveness is evidei-xed by the number oi 
survey sites (almost half) which reported involvement 
in new follow-up activities and plans. But the 
majority of SDAs seem uncertain how hest to structure 
such efforts and use the information thty yield. 



2) Strategies for Identifying Causes of Poo r 
Performance (4.3) " 

All local constituencies tend to place some premium on 
evaluation approaches which offer the potential for 
identifying and predicting factors associated with poor 
performance. Only local elected officials, however, 
gave this evaluation element a top mean rating (5). 
Current SDA systems are typically geared to pick up 
indications of possible poor performance results fairly 



early in the program year. But administrative staff 
often have a difficult time determining the Lctual 
causes or conditions responsible for poor performance 
showings and ameliorating these conditions. Staff have 
even less ability to predict in advance which 
organizational or programmatic variables are most 
crucially linked with performance results. 



3) Strategies for evaluating program effectiveness and 
benefits from an employer perspective (4.2) 

This evaluation element tended to be rated somewhat 
higher in importance by local elected officials and PIC 
members than by SDA respondents. SDA respondents do 
place a relatively high priority on evaluation 
information that could be used for employer marketing. 
But some respondents believe that attempts to evaluate 
program impact on employing firms would be impractical 
and would prove less useful than qualitative feedback 
on employer satisfaction. SDAs could use assistance in 
techniques for tapping and analyzing employer feedback 
and using this information in marketing. 

4) Techniques for analyzing the relationship between 
program strategies and performance results (4) 

No SDA can afford to ignore performance standards and 
no SDA director wants to be stuck with planned 
performance objectives which can't possibly be met 
through the mix of program strategies or services that 
have been funded. SDA respondents place a high premium 
on evaluation designs which offer mechanisms (such as 
linear projection models) for predicting the affect 
that service mix, targeting strategyt and other program 
dcisign decisions are likely to have on performance 
results. These kinds of approaches, while not seen as 
infallible, are viewed as giving administrative 
entities greater control over performance outcomes. 




5) Practicable approaches for evaluating the benefits 
o f JTPA for the community (3.9) 

All of the groups surveyed tended to rate the ability 
to evaluate and demonstrate the benefits of JTPA for 
the community as being an, at least, moderately 
important element in an overall SDA evaluation design. 
Mean ratings for this evaluation element were highest 
for PIC members and lowest for Washington SDA 
A espondents. 

If;? 



No one we surveyed felt that a net impact evaluation 
model was practicable at the SDA level. (One SDA, 
however, was establishing the kind of linkages to 
automated U.I. and welfare data bases which would make 
some form of net impact assessment feasible.) But 
respondents did feel it was important to demonstrate 
JTPA's effectiveness as an investment strategy and its 
beneficial effects for the community at large. 

6) Approaches that allow for assessment of the 
effectiveness of a specific program strategy or 
component (3,8) 

Respondents felt that an overall evaluation design 
package tailored to SDA-level use should include 
approaches for the intensive assessment of a specific 
strategy, component, or project. SDAs tend to focus 
evaluations geared toward program upgrading or redesign 
on only specific elements of the system; rarely is the 
total system up for grabs at any given moment in time. 
Process evaluation models which combine qualitative and 
quantitative techniques are viewed as being appropriate 
for intensive single- component assessments at the SDA 
level . 



7) Techniques for comparin;^ program strategies and 
results across contractors (3.4) 

In a climate that places a high premium on performance 
but offers a declining base of resources for funding 
program services, valid techniques for rating 
contractor effectiveness are increasingly important at 
the SDA level. Larger systems with a number of 
competitive contractors tend to operate in a political 
environment which focuses considerable scrutiny on 
funding decisions. Most respondents realized that 
comparison of performance across contractors should 
take into account the characteristics of those being 
served and the types of services being provided. 

8) Techniques for evaluating local processes -> such as 
planning t managing and contracting approaches ~ and 
assessing the affects of these processes on program 
design and quality (3.3) 

Among the groups surveyed, local elected officials and 
Washington SDA respondents were more apt to place a 
relatively high degree of importance on this element of 
local-level evaluation. Several SDAs in the national 
sample reported that they had conducted limited 
evaluations of specific local processes which had 



produced useful information. But other SDA respondents 
felt that self-evaluation of this type was apt to be 
less than objective, and most preferred to focus their 
limited evaluation capabiliti es on training strategies 
and program outcomes. 

9) Approaches for evaluating the effectiveness of 
coordination linkages with non-JTPA programs (2,9) 

Among all the groups surveyed, only local elected 
officials and PIC members rated the assessment of 
coordination linkages as a relatively important element 
in an SDA-level evaluation design package. Most SDA 
respondents had a hard time envisioning practical 
evaluation approaches which might yield useful insights 
on the effectiveness of coordination efforts. And a 
number of respondents indicated that meaningful 
coordination with non-JTPA programs was more dependent 
upon action taken at the state level rather than the 
local level . 

10) Ability to analyze SDA effectiveness in comparison 
with other SDA systems 

We did not explicitly ask for a rating on this 
potential feature of an overall evaluation design 
package. But many respondents mentioned that they 
would welcome the adoption of models which allow some 
ability to compare their effectiveness in various areas 
with that of other SDAs. Several respondents mentioned 
that the validity of such comparisons would depend upon 
a uniform base of definitions for key reporting 
categories, such as "enrollment" and "plac lament" . 
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SUGGESTIONS FOR EVALUATION MODEL DESIGNERS 



we asked survey respondents to tell us (on an 
open-ended basis) what they felt were the most 
important considerations evaluation designers should 
bear in mind in developing models or guides which would 
be useful at the SDA level. Their responses were 
recorded, grouped, and analyzed. A summary of 
suggestions from local constituencies is offered below. 



1) Focus on specific purposes or uses 

Local constituencies are not likely to adopt evaluation 
models proposed by outside designers without a clear 
sense of how the outputs of suggested evaluation 
designs can be used at the local level. A number of 
respondents suggested that evaluation guides begin with 
specific purposes or local utilization uses (such as 
better targeting of program services); indicate the 
kinds of evaluation information needed to effectively 
address these purposes; and then go on to suggest 
appropriate methodologies for securing and analyzing 
evaluation information. 

Evaluation models which are capable of generating 
findings useful for a variety of different purposes 
will clearly be preferred. Specifically, local 
constituencies will want to know how proposed 
evaluation models will be of potential assistance to 
them in the following areas: 1) managing system 
performance; 2) developing local policy goals; 3) 
allocating resources; 4) designing, developing, and 
directing programs; 5)" publically documenting system 
accomplishments; and 6) marketing the program to local 
employers . 
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2) Make guides as practical. simple. a nd clea r as 

possible ' 

In preparing written guides, designers should offer a 
clear explanation of the model??, their key elements, 
uses, and limitations. Designers should not assume a 
high level of technical expertise across the SDA 
audience. Materials should emphasize practical 
considerations and be written in s straight forward and 
noncondescending style. Even SDA r epresentatiY«:^ virh 
a relatively high degree of sophistication in 
evaluation methodologies will be put off by guides 
which appear to be highly academic and too far rerccved 



from the program environment. The guides should be 
understandable to administrators and policymakers and 
useful to staff with various functions in the SDA 
system - planners, managers, fiscal/MIS staff - as well 
as evaluation specialists. The guides should clearly 
Indicate those areas of design and implementation where 
special technical expertise or outside guidance is 
advisable. 



3) Encompass a range of design options 

SDIs vary widely in regard to local program focus, 
organizational and service delivery structures, and 
system capabilities. If the models are to be widely 
used at local level, they must include options for 
tailoring evaluation designs to local circumstances , 
capabilities, and needs. Specifically, respondents 
expressed major concern that some options be provided 
that ar-^ realistic for SDAs with small grants and very 
limited staff capabilities. 

While local constituencies expressed some concern for 
validity and reliability of results, they are more 
Interested in capturing rich and useable feedback than 
in conducting rigorously scientific evaluation,^. SDA 
constituencies are wary of models which require 
experimental controls or compl^*x sampling designs. 
They would like more practical options for collecting 
and analyzi'^g feedback on program effectiveness and 
longer-ter- outcomes . They would like the design 
flexibility to combine both qualitative and 
quantitative approachesa They would also like design 
packages:: which offer the f 1 exibility to focus 
intensively on single program components or projects or 
to .duct broader systemwide assessments^ to conduct a 
quick limited-purpose assessment or to incrementally 
impl'^^ent a more extensive ongoing evaluation system. 



^ ' C^^. *^i der bot h n ational perform a nc e standards and 

SDA coiistituencies are concerned vith analyzing and 
tracking t^ystem performance in regard to the national 
;srformance standards applied by the state in awarding 
performance incentive funds. They are also interested 
in evaluating the extent to which programs are 
achieving local policy goals. Models for ongoing 
evaluation initiatives at thz local level should 
encompass the "uniform" performance standard indicators 
(including regression model adjustments ? but they 
should also be flexible enough to allow SDAs to focus 
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specifically on those goals of particular concern to 
local policymakers • 

5) Back up models with technical assistance 

Respondents recognize that there is a limit to the 
extent to which national guides can offer technical 
advice geared to local-level adoption and 
implementation of the models« Manj respondents felt 
that the prospects of local SDAs actually using the 
suggested models would be greatly enhanced by a focused 
technical assistance and training effort. 

SDA-level actors view evaluation initiatives with both 
interest and trepidation. Less sophisticated staff are 
particularly fearful of the potential technical 
difficulties involved in evaluation. Most SDA 
constituencies will need to be convinced that 
implementation of proposed evaluation approaches is 
both technically feasible and likely to produce outputs 
which are worth their efforts. Several respondents 
suggested regional workshops or on-site training 
provided by persons with actual experience in 
conducting successful evaluation initiatives at the 
local level . 
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LOCAL SYSTEM CAPABILITIES AND CONTINGENCIES 



1) Data Collections R etrieval, and Analysis 

^llt.l^ ^mU^^^^ surveyed have a management information 
system (MIS) which tracks program transactions 
teJmii'?'"^'* training assignments, completions? 
terminations, and placement outcomes) and a variety of 
ni pre-program status measures 

on an individual participant basis. Most SDAs can 
break out participant-based performance indicators by 
major component or contractor on a monthly basis. 

All but two of the SDAs operate Separate systems for 
tracking financial transactions. While these systems 
allow for analysis of expenditures by major component 
or contractor, it is often difficult for SDAs to 
accurately compute indicators which integrate 
participant and financial data (such as cost per 
participant and cost per entered employment) on a 
current basis. Lack of integration in regard to 
financial and MIS data is viewed as a major limitation 
in local evaluation efforts by over one-third of the 
bDAs surveyed. 

The majority of the SDAs (all of the Washington SDAs 
and over 60 percent of the national sample) are usine 
management information systems which have been, or are 
in the process being, adopted on a statewide basis. 
All but one SDA will be tied into a statewide MIS by 
the end of the next program year. About one-quarter of 
the national sample SDAs were using the Washington 
State data flex software. 

The hardware used for SDA MIS systems included- 
mainframe computers (used by a regional network of SDAs 
?5 ° municipal system) (2 SDAs); mini computers 

U bDAs); micro computers (15 SDAs); and "dumb" 
terminals tied into a statewide computer (2 SDAs) 
Only one SDA was currently using a manual (card sort) 



The SDAs most likely to be fully satisfied with their 
MIS capabilities were those using systems developed 
especially for local level use, which had a good deal 
of flexibility in data base management, retrieval and 
analysis. (These generally required mini or mainframe 
capabilities . ) 

SDAs using state-developed MIS packages with micro 
computers or dumb terminals were apt to complain that 



these systems were geared toward meeting state-level 
reporting requirements, rather than meeting local 
information and analysis needs. But those SDAs which 
had enhanced their systems with added program 
capabilities were, by and large, satisfied with the 
statewide system. 

Only three SDAs reported that they were currently 
experiencing no problems with their MIS. The most 
commonly reported problems were: 1) lack of ability to 
break out needed data at a sufficient level of detail; 
2) need for manual transcription/computation of plan vs 
performance indicators; 3) not being programmed for 
performance standard calculations; 4) lack of staff 
expertise in MIS operation and programming; and 5) 
system bugs and breakdowns. The majority of the SDAS 
were adjusting to new MIS systems. Many said they were 
still experimenting and had not yet fully tapped their 
systems' capabilities. 

When asked how well equipped their MIS was to support 
evaluation activities currently underway or planned for 
the future, one-eighth of the SDA respondents rated 
their MIS capabilities as excellent. Three-fourths of 
the SDAs reported that their basic systems were at 
least adequate - though about half felt that 
adjustments and additional programming expertise would 
be required. Only four SDAs rated their systems as 
poor or inadequate for supporting local evaluation 
initiatives (and two of these SDAs expected to change 
their system during the coming program year.) 

Most of the SDAs surveyed collect 30 day retention data 
on placements, which is integrated in the KIS data 
base. About one-quarter of the SDAs currently collect 
no additional follow-up data. About two-third collect 
or receive follow-up data on (at least a sample of) 
participants at 12 to 13 weeks following placement or 
termination. Only two of SDAs were currently 
collecting longer-term (4-12 months) follow-up data. 

Most of the current follow-up systems have been 
initiated to meet state requirements, rather than to 
serve local purposes, according to SDA respondents. 
Only one-quarter of the SDAs are currently conducting 
locally designed follow-up efforts. Locally designed 
designed and initiated follow-ups are more likely to 
include employer-based as well as participant based 
contacts and to solicit qualitative feedback as well as 
objective data on employment retention and earnings. 
Only one SDA has a follow-up system which collects U.I. 
wage reporting data and welfare payments data both pre 
and post program terT)ination . 



ollow-up is a relatively new effort for most SDAs. 
Close to half of those surveyed indicated that they 
planned either to initiate new local follow-up designs 
or to expand, augment or improve existing follow-up 
efforts during the next six months. Only two of the 
SDAs were currently integrating follow-up data in the 
nib data base, but of number of others said they 
planned to do so. 

A number of SDAs reported as major evaluation 
constraints their inability to access information from 
the Employment Service (on participant ep.rnings or 
employment status) or the welfare department (on 
welfare payments to participants.) Few SDAs were 
attempting to measure pre/post program increases in 
earning or employment stability or reductions in 

^td!^'^^**^^^'"^"*^^* respondents felt that state 

JTPA officials should do more to expedite the release 
of such information. 

In the "data block" environment, the Denver information 
sharing system constitutes a notable exception. 
Colorado is a wage reporting state, and the Denver SDA, 
which already has access to some U.I. and welfare data, 
will soon have the capability to directly interface 
with U.I. and welfare data bases, access needed pre and 
post program information on JTPA participants, and 
store this information in its own fully automated data 
base . ' 



2) Funding Evaluation Initiatives 

Scarcity of funds for conducting evaluation was viewed 
as a major obstacle by over two-thirds of the SDA 
respondents. SDAS with smaller II-A grants (under $2 
million) found it especially difficult to set aside 
funds for conducting program evaluation activities when 
administrative budgets were already stretched to the 
limit. Even SDAs with larger II-A grants, greater 
administrative budgeting flexibility , and a historv of 
substantial evaluation activity, reported considerable 
difficulty in breaking out funds for new evaluation 
initiatives. 

Evaluation activities are often tied into other 
functions of the administrative entity, and Auy SDA 
respondents have some difficulty estimating disllar 
expenditures for specific evaluation initiatives. 
There is clearly a vide range of variation in ♦•he 
amounts that individual SDAs have expended on 
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evaluation to date. Larger SDAs raported expenditures 
from annual administrative budgets of $15,000 to 
$50,000 to support ongoing follow-up efforts A 
smaller SDA reported spending $6,000 on an assessment 
of current program effectiveness in serving 
harder-to-serve target groups. Another SDA of about 
the same size spent only 80 man hours (about $1,000) on 
an assessment of an institutional skills training 
project. * 

Some SDAs have found creative ways to fund follow-up 
efforts and other evaluation initiatives. One SDA 
director was successful in enlisting the cooperation of 
business school staff from a local university in 
developing a design for program performance review and 
follow-up and using graduate students to conduct the 
study. Another SDA with a Title II-A grant of less 
than $1 million got assistance from the National 
Alliance of Business in designing and conducting a 
program effectiveness assessment, with $3,000 in SDA 
resources matched by a $3,000 grant from the state. 

Resource limitations are a major consideration in the 
design and implementation of evaluation activities, but 
i^.^f clear that scarcity of funds need not preclude 
SDA-level evaluation. 
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3) Staffing Evaluation Initiatives 

Capabilities for designing and staffing local 
evaluation initiatives vary widely among the SDAs 
surveyed. While one of the largest SDAs has over 60 
persons on staff (some of which are involved in service 
delivery,) several of the smaller SDAs have only 2-3 
staff persons in the entire administrative entity. 

The bulk of SDA-level staff are deployed in contracts 
management, financial accounting, MIS, and PIC 
support. Only a handful (3 of the 23) of the SDAs we 
surveyed have one or more staff persons designated a^; 
evaluation specialists, and these are all larger 
systems. 

While relatively few (less than 20 percent) of the SDAs 
surveyed admitted to a serious lack of local expertise 
for conducting evaluation, it is clear that much of the 
local evaluation activity is carried out by generalists 
or others with limited technical background in 
evaluation. 

In about one-third of the SDA systems, other staff - 
MIS specialists, planners, program monitors. 



administrators - are charged with responsibility for 
evaluation^related activities. A number of respondents 
feit that there are clear advantages to allocating 
evaluation roles across various staff units, rather 
than vesting evaluation functions in an independent 
unit removed frbiii ongoing system management and 
planning. But in over half of the SDAs surveyed, no 
one on ths staff is charged with ongoing responsibility 
for program evialuation. 



At least half of the SDAs that reported conducting 
follcw-up contacts, intensive program reviews, or 
special assessment studies, have relied heavily on 
outside assiistahciBi Outside resources used by SDA 
respondents included paid consultants, public interest 
orgahiizations, student interns, university staff and 
graduate students, and, in several cases, program 
operatots. 



^) PIC Roles S Support for Evaluation 

In the majority (two-thirds) of the SDAs surveyed, the 
Private Industry Council has a program evaluation or 
oversight committee which meets on at least a quarterly 
basis. In all of the SDAs, information on system 
performance is presented to the PIC on at least a 
quarterly basis. The emphasis to date in PIC oversight 
roles has been on performanca monitoring, rather than 
effectiveness evaluation. 

In most cases PIC members review plan vs performance 
statistics prepared by SDA staff, respond to 
performance problems redflagged by staff, and initiate 
or approve corrective action strategies. PIC Members 
are ikely to become more substantivly involved in 
assessment issues through participation on program 
plai^ning or funding committees, or subcommittees that 
focus on specific program components or issues. 

PIO members would like better information for policy 
development, program funding, and design decisions, but 
they lack any clear picture of just how local 
evaluation efforts could improve the base of 
intormation and understanding. They tend to equate 
evaluation with statistical reports, and many PIC 
members ere already feeling overloaded by the reams of 
performance data currently being generated and 
dissen^nated at the SDA level. As one PIC member 
'jointed oi:t: "There's a limit to how much any 
volunteer can read or digest on a regular basis. We 



don^t need more information; what we need is better 
analysis." 

PIC members will generally support special purpose 
assessments when there is a need for specific types of 
information • But SDA staff report chat it is harder to 
gain support for sustained ongoing evaluation 
initiatives. Attempts to promote new evaluation 
initiatives that require substantial SDA- level 
investments must specifically address the concerns of 
PIC policymakers • PIC members will want to know how 
proposed evaluation models will assist them in policy 
development , planning, funding, end oversight, and in 
improving the quality of local programming. 
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